Bug 1073 - Microwatt verilator sim - setting up chroot and documentation
Summary: Microwatt verilator sim - setting up chroot and documentation
Status: RESOLVED FIXED
Alias: None
Product: Libre-SOC's first SoC
Classification: Unclassified
Component: Source Code (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- enhancement
Assignee: Andrey Miroshnikov
URL:
Depends on:
Blocks: 961 1086
  Show dependency treegraph
 
Reported: 2023-04-27 16:35 BST by Andrey Miroshnikov
Modified: 2023-10-11 15:18 BST (History)
4 users (show)

See Also:
NLnet milestone: ---
total budget (EUR) for completion of task and all subtasks: 0
budget (EUR) for this task, excluding subtasks' budget: 0
parent task for budget allocation:
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Miroshnikov 2023-04-27 16:35:53 BST
I've setup a new Libre-SOC wiki page documenting the exact order of instructions I followed to set up a new chroot for working with Microwatt for documentation and reproducibility.
Currently stumbling on the same issue I've had back in January
https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47
https://libre-soc.org/irclog/%23libre-soc.2023-01-27.log.html#t2023-01-27T05:30:02
https://libre-soc.org/irclog/%23libre-soc.2023-01-30.log.html#t2023-01-30T10:05:14

-Libre-SOC wiki page: https://libre-soc.org/HDL_workflow/microwatt_tutorial/


The Libre-SOC microwatt tutorial shows the following command to time how long the sim takes to run:
(microwatt):$ time ./microwatt-verilator

No additional arguments to the binary are specified.

As I've covered on the wiki page, the hello_world binary is loaded into RAM at synthesis, so compiled verilator sim should already have the program loaded.
In the Makefile,
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;h=610f48d8c89be6d5b9902d7f1bf61f8b6d98ffc0;hb=refs/heads/verilator_trace
`RAM_INIT_FILE` (line #144) is specified as `hello world.hex` (the Microwatt 'light bulb' example code).
The Makefile uses this `RAM_INIT_FILE` argument for generating the `microwatt.v` verilog file (line #249).
Finally, to generate the `microwatt-verilator` binary, `microwatt.v` is pulled in as a dependency (line #254).

However, even after running for over 18min on a Ryzen 3600 and verilator set to 11 threads, no terminal output appears.


Please help me understand what I'm missing.


What would really help is some way to measure the progress of the simulation (and that it's not just sitting, running NOPs).

(This bug might be in the wrong category, perhaps documentation is better?)
Comment 1 Luke Kenneth Casson Leighton 2023-04-27 16:41:55 BST
(In reply to Andrey Miroshnikov from comment #0)

> As I've covered on the wiki page, the hello_world binary is loaded into RAM
> at synthesis, so compiled verilator sim should already have the program
> loaded.

mmm.... should be ok.

> However, even after running for over 18min on a Ryzen 3600 and verilator set
> to 11 threads, no terminal output appears.

don't do that arbitrarily.  you need to enable the FST compile option
which outputs gtkwave traces then go look at them.

> What would really help is some way to measure the progress of the simulation
> (and that it's not just sitting, running NOPs).

gtkwave trace, as i mentioned on the last call.
Comment 2 Andrey Miroshnikov 2023-04-27 23:01:40 BST
After our conversation on IRC:
https://libre-soc.org/irclog/latest.log.html#t2023-04-27T16:38:23

(In reply to Luke Kenneth Casson Leighton from comment #1)
> don't do that arbitrarily.  you need to enable the FST compile option
> which outputs gtkwave traces then go look at them.
> 
> > What would really help is some way to measure the progress of the simulation
> > (and that it's not just sitting, running NOPs).
> 
> gtkwave trace, as i mentioned on the last call.

I was able to convert using vcd2fst and open the .fst file in gtkwave. The problem seems to be with the core being idle.

Looking at the bram.dump in the hex editor, I see these:
pc         ff000000 insn ffffffff msr 8000000000000001
pc         ff000004 insn ffffffff msr 8000000000000001
pc         ff000008 insn ffffffff msr 8000000000000001
                          rd @ 00000100 di                0 sel ff ........
                          rd @ 00000101 di                0 sel ff ........
And so on...
Looks like PC is starting at 0xff00_0000, which follows the soc.vhdl and microwatt.h definitions for DRAM init.


I found the arguments you added for using the BRAM dump or verilator.save files:
-s FILENAME
Example: ./microwatt-verilator bram.dump
Loads a previous dump of the BRAM (which allows you to resume from where you left off)

+[TICKNUMBER]
Example: ./microwatt-verilator +1999990
Loads from save called "verilator.save.1999990".

However it took me a while to finally understand how arguments are passed to verilator itself (using the +verilator prefix).

Found the list of runtime verilator args that can be passed: https://verilator.org/guide/latest/exe_sim.html

None of those seem relevant. Do I need point the sim to a starting address, or do I need to supply a binary instead?

I would've thought that if the microwatt reset address is set to where the code is located, it should begin executing.

One more thing I tried looking before finishing for today is looking at the ghdl synthesis in the Makefile.
On line #233 and #235 of the Makefile, the (generic?) options -gRAM_INIT_FILE as well as -gRESET_ADDRESS specify the RAM init file to be passed to ghdl, however I haven't found any documention on these options at all.
Comment 3 Luke Kenneth Casson Leighton 2023-04-27 23:18:37 BST
(In reply to Andrey Miroshnikov from comment #2)

> I would've thought that if the microwatt reset address is set to where the
> code is located, it should begin executing.

indeed - but if you've compiled it to be at a specific memory address,
and you "load" it into a different one, where do you think the PC will
go on the very first branch?
Comment 4 Jacob Lifshay 2023-04-27 23:19:54 BST
(In reply to Luke Kenneth Casson Leighton from comment #3)
> (In reply to Andrey Miroshnikov from comment #2)
> 
> > I would've thought that if the microwatt reset address is set to where the
> > code is located, it should begin executing.
> 
> indeed - but if you've compiled it to be at a specific memory address,
> and you "load" it into a different one, where do you think the PC will
> go on the very first branch?

to the correct spot since nearly all branches are PC-relative? :P
Comment 5 Jacob Lifshay 2023-04-27 23:22:19 BST
(In reply to Jacob Lifshay from comment #4)
> (In reply to Luke Kenneth Casson Leighton from comment #3)
> > (In reply to Andrey Miroshnikov from comment #2)
> > 
> > > I would've thought that if the microwatt reset address is set to where the
> > > code is located, it should begin executing.
> > 
> > indeed - but if you've compiled it to be at a specific memory address,
> > and you "load" it into a different one, where do you think the PC will
> > go on the very first branch?
> 
> to the correct spot since nearly all branches are PC-relative? :P

generally embedded code needs to know which address it's at, so all the non-pc-relative things work correctly
Comment 6 Luke Kenneth Casson Leighton 2023-04-28 00:39:27 BST
(In reply to Jacob Lifshay from comment #5)

> generally embedded code needs to know which address it's at, so all the
> non-pc-relative things work correctly

such as "load the immediate address of a bootloader magic constant".

you'll find that in the Makefile there is a #define of the offset
address, that *absolutely must* be encoded correctly to the
address that you expect the binary to be uploaded to.  this #define
goes into the linker script via some makefile target voodoo involving
gcc -E (i think - something like that) which can do macro-substitution.
Comment 7 Andrey Miroshnikov 2023-04-28 15:57:02 BST
(In reply to Andrey Miroshnikov from comment #2)
> On line #233 and #235 of the Makefile, the (generic?) options
> -gRAM_INIT_FILE as well as -gRESET_ADDRESS specify the RAM init file to be
> passed to ghdl, however I haven't found any documention on these options at
> all.

Figured out that "-g" prefixed arguments allow to pass parameters into the VHDL module, so that's where RAM_INIT_FILE, and RESET_ADDRESS arguments go to. Perhaps I should do a basic ghdl counter tutorial, like we have with nmigen, so that devs actually know how to work with this. Even if we don't use VHDL ourselves, still need to know how to access it for Microwatt.


(In reply to Luke Kenneth Casson Leighton from comment #6)
> such as "load the immediate address of a bootloader magic constant".
> 
> you'll find that in the Makefile there is a #define of the offset
> address, that *absolutely must* be encoded correctly to the
> address that you expect the binary to be uploaded to.  this #define
> goes into the linker script via some makefile target voodoo involving
> gcc -E (i think - something like that) which can do macro-substitution.

Searching for the "bootloader magic constant"...
In the Makefile for hello_world, there are no address offset parameters.
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=hello_world/Makefile;h=dd51bd38f9ae428328cb8f0d2c3eea479068d997;hb=refs/heads/verilator_trace

Makefile provides a RESET_ADDRESS of 65280 (0xff00), because VHDL doesn't allow passing integers bigger than 2^32 apparently. Then in top-generic.vhdl:
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=fpga/top-generic.vhdl;h=0ce8456d0f781e858a4d70c8ffe46b90c119d216;hb=refs/heads/verilator_trace#l108
This integer is converted to an unsigned 48-bit number, and concatinated with lower 16-bits. This gives the correct reset address of 0xff00_0000.

The RAM_INIT_FILE (in my case the hello_world binary) is passed to the soc.vhdl, and then to wishbone_bram_wrapper.vhdl.
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=soc.vhdl;h=b02fe12c6b9d2120c759c6bb3886bb8d6ad917ee;hb=refs/heads/verilator_trace
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=wishbone_bram_wrapper.vhdl;h=4e3fb506dfecb66bb6bf165251957b5d2547f26d;hb=refs/heads/verilator_trace
Inside wishbone_bram_wrapper.vhdl, the main_bram module (with RAM_INIT_FILE) is only generated if the SIM_MAIN_BRAM parameter is true. The Makefile sets it to false by default, which would explain with the RAM is not actually initialised.

Setting SIM_MAIN_BRAM to true in the Makefile causes an error during "make microwatt-verilator":
ERROR: vhdl import failed.
make: *** [Makefile:252: microwatt.v] Error 1

Earlier, this error occurs about multiple assignments in wishbone_bram_wrapper.vhdl
wishbone_bram_wrapper.vhdl:23:9:error: multiple assignments for "wishbone_out" offsets 0:63
        wishbone_out : out wishbone_slave_out;

I'm guessing I don't need this SIM_MAIN_BRAM parameter, why else would it be disabled.


Another parameter I tried to change (after reverting SIM_MAIN_BRAM to false) is SIM_BRAM_CHAINBOOT. In the Makefile it is set to 6291456 (0x600000, needed for the linux image).
Setting it to 0, I get the following error:
error: override for generic "sim_bram_chainboot" is out of bounds
ERROR: vhdl import failed.
make: *** [Makefile:251: microwatt.v] Error 1

In soc.vhdl line #62, the default value is set to 0, so I don't know why it complains about being out of bounds.


Other than this, I have not found any other offset values to adjust in the Makefile for microwatt-verilator.

microwatt_soc.h (used for compiling PowerISA binaries to run) has some potentially relevant parameters:
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=include/microwatt_soc.h;h=989721b14e57781c3099b10936e6f91c7b69c767;hb=refs/heads/verilator_trace
MEMORY_BASE=0x0000_0000 /* "Main" memory alias, either BRAM or DRAM */
DRAM_BASE=0x4000_0000 /* DRAM if present */
BRAM_BASE=0x8000_0000 /* Internal BRAM */
Also there's an option if STANDALONE_MINI_BIOS is defined:
DRAM_INIT_BASE=0x0000_0000 /* alternative, for verilator simulation */
Otherwise
DRAM_INIT_BASE=0xff00_0000 /* Internal DRAM init firmware */

Also, after looking at IRC logs:
https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html#t2022-01-26T19:24:55
Saw that a binary can be passed in to microwatt-verilator:
./microwatt-verilator tests/hello_world/hello_world.bin

However I'm still missing the correct offset setting, or something like that.
The bram.dump first few lines:
pc         ff000000 insn ffffffff msr 8000000000000001
pc         ff000004 insn ffffffff msr 8000000000000001
pc         ff000008 insn ffffffff msr 8000000000000001
                          rd @ 00000100 di         48000000 sel ff ...H....
                          rd @ 00000101 di                0 sel ff ........

Not sure where to go from here.

Is there a way to search for a string of text in all the email and IRC archives? Your previous updates from 2021-2022 on Microwatt would be handy, and I'd prefer not to go through months of IRC logs and emails by hand, if there's a way to search it.
Comment 8 Luke Kenneth Casson Leighton 2023-04-28 16:37:00 BST
ok you've misunderstood: the *executable binary* you have compiled with
an incorrect offset.  check the linker script.
Comment 9 Andrey Miroshnikov 2023-04-28 22:14:28 BST
(In reply to Luke Kenneth Casson Leighton from comment #8)
> ok you've misunderstood: the *executable binary* you have compiled with
> an incorrect offset.  check the linker script.


https://git.libre-soc.org/?p=microwatt.git;a=blob;f=hello_world/powerpc.lds;h=96bc6b9b16120335e73f6e90c29b3573d2c55e59;hb=refs/heads/verilator_trace

I'm assuming "_start = .;" is the entry point, and the address is equal to the first available (0x0).

Using objdump, viewed the regions of the elf file before changing anything.
objdump -x hello_world.elf <- shows all headers
objdump -h hello_world.elf <- shows section headers

-h shows the section list (only first three shown):
Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .head         00000f84  0000000000000000  0000000000000000  00010000  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .sfpr         00000014  0000000000001000  0000000000001000  00011000  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  2 .text.startup.main 00000070  0000000000001014  0000000000001014  00011014  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
                  
-x showed the symbol table, and the main code seems to be where .text.startup.main is:
0000000000000000 l    d  .head	0000000000000000 .head
0000000000001000 l    d  .sfpr	0000000000000000 .sfpr
0000000000001014 l    d  .text.startup.main	0000000000000000 .text.startup.main
...
0000000000000000 g       .head	0000000000000000 _start
0000000000001014 g     F .text.startup.main	0000000000000070 0x60 main

which is why I tried setting the _start to 0x1014.

Tried changing _start to:
- 0x1000 (where .text section begins)
- 0xff00_1000 (my guess what the SoC level address would be)
- 0x1014
- 0xff00_1014

The last one I tried (_start = 0xff001014;) changed the bram.dump slightly, however as before (and what I didn't show from previous bram.dump, sorry) is that the PC is stuck at 0x800:
pc         ff000000 insn ffffffff msr 8000000000000001
pc         ff000004 insn ffffffff msr 8000000000000001
pc         ff000008 insn ffffffff msr 8000000000000001
                          rd @ 00000100 di         48000000 sel ff ...H....
                          rd @ 00000101 di                0 sel ff ........
                          rd @ 00000102 di                0 sel ff ........
                          rd @ 00000103 di                0 sel ff ........
pc              800 insn 48000000 msr 8000000000000001
                          rd @ 00000104 di                0 sel ff ........
                          rd @ 00000105 di                0 sel ff ........
                          rd @ 00000106 di                0 sel ff ........
pc              800 insn 48000000 msr 8000000000000001
                          rd @ 00000107 di                0 sel ff ........
pc              800 insn 48000000 msr 8000000000000001

I'm not really sure how the PC goes back from ff00_0008 to 0x800.
Comment 10 Luke Kenneth Casson Leighton 2023-04-28 23:25:01 BST
(In reply to Andrey Miroshnikov from comment #9)

> I'm not really sure how the PC goes back from ff00_0008 to 0x800.

an exception.  look up the table somewhere...  page 1067 rings 
bell somewhy.
Comment 11 Andrey Miroshnikov 2023-05-01 14:48:32 BST
(In reply to Luke Kenneth Casson Leighton from comment #10)
> an exception.  look up the table somewhere...  page 1067 rings 
> bell somewhy.

Found the table you mentioned:
PowerISA v3.1B pdf:
Section 7.5 Interrupt Definitions

PDF page 1302 (spec page 1276)
Figure 70. Effective address of interrupt vector by
interrupt type:
...
00..0000_0800 | Floating-Point Unavailable
...

So I'm guessing this is something about floating-point hardware not being present?

Looking at the bram.dump line in detail:
pc ff000000 insn ffffffff msr 8000000000000001

Does it mean:
PC: ff00_0000
decoded instruction: ffff_ffff (I'm guessing this is incorrect decoding, because the word is not an instruction but data, or empty)
MSR value: 8000000000000001
Machine State Register (Book III, Section 4.2.1):
bit0: 0 (32-bit mode) / 1 (64-bit mode)
... (ignoring the rest)
bit63: 0 (big-endian) / 1 (little-endian)

So just out of reset the processor is:
- in 64-bit thread mode, little-endian
- PC at ff00_0000
- Current instruction is ffff_ffff (not a valid instruction?)

It looks like (as you have pointed out already), the processor is expecting instructions at ff00_0000 (reset address), but the program binary is not there.
It goes through addresses ff00_0000, ff00_0004, ff00_0008, but continues to find no instructions.

What do these statements mean:
rd @ 00000100 di         48000000 sel ff ...H....

Read value at address 0000_0100 is 4800_0000?


Sorry for being dumb, I think I understand the problem, but I don't know how to go about pointing the cpu to the right place.
Comment 12 Luke Kenneth Casson Leighton 2023-05-01 15:05:05 BST
(In reply to Andrey Miroshnikov from comment #11)

> Sorry for being dumb, I think I understand the problem, but I don't know how
> to go about pointing the cpu to the right place.

i already said: you don't.  you compile the *binary* to be at
the correct place and place the *binary* at that memory location
by specifying the correct options to ls2 to place the *binary*
at that address on the wishbone bus.
Comment 13 Luke Kenneth Casson Leighton 2023-05-01 15:11:54 BST
lkcl@fizzy:~/src/libresoc/microwatt2$ git diff
diff --git a/Makefile b/Makefile
index 610f48d..17d1548 100644
--- a/Makefile
+++ b/Makefile
@@ -214,10 +214,10 @@ _fpga_files = fpga/soc_reset.vhdl \
        nonrandom.vhdl
 
 # use an alternative core (in verilog)
-EXTERNAL_CORE=false
+EXTERNAL_CORE=true
 # VHDL does not allow integers greater than 2^32, so shift down
 # by 16 bits and add 16 bits zeros back on in soc-generic.vhdl
-RESET_ADDRESS=65280 # 0xff00_0000>>16
+RESET_ADDRESS=0 # 65280 # 0xff00_0000>>16
Comment 14 Andrey Miroshnikov 2023-05-02 01:07:08 BST
IRC logs from yesterday:
https://libre-soc.org/irclog/latest.log.html#t2023-05-01T15:07:37

(In reply to Luke Kenneth Casson Leighton from comment #13)
> -RESET_ADDRESS=65280 # 0xff00_0000>>16
> +RESET_ADDRESS=0 # 65280 # 0xff00_0000>>16

As Luke has suggested:
https://libre-soc.org/irclog/latest.log.html#t2023-05-02T00:30:56

Thanks to clearing up what I was actually doing (running Microwatt by itself and *not* in the ls2 fabric), and going ahead with changing the RESET_ADDRESS parameter to 0 in:
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;h=610f48d8c89be6d5b9902d7f1bf61f8b6d98ffc0;hb=refs/heads/verilator_trace#l220

fixed the problem with verilator, and hello_world ran.

In the IRC chat Luke also mentioned a makefile to automatically generate the correct powerpc.lds with the appropriate starting address, found in ls2 repo:
https://git.libre-soc.org/?p=ls2.git;a=blob;f=hello_world/Makefile;h=50f039112f54165f8f6f7421ac62be1661889576;hb=HEAD#l9

Later today, I will test this with Microwatt to make sure the original RESET_ADDRESS 0xff00_0000 works, update the wiki tutorial, and see about recording a short video to demonstrate setting up, and running code.
Comment 15 Andrey Miroshnikov 2023-05-08 20:55:58 BST
After getting the sim to run last week, I forgot that the vcd trace file was empty.

Luke helped out on IRC:
https://libre-soc.org/irclog/%23libre-soc.2023-05-07.log.html#t2023-05-07T14:42:12

I discovered the reason: the tick() function has a boolean 'dump' argument which, if set to true, will dump the trace data for the current clock cycle. The reason it was dumping the first time 0, is because during reset condition, tick() dump argument is hard-coded to true:
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l261

The reason there was no further dumping, is because the TRIGGER_ENABLE define was causing the 'traceme' dump flag to be set to false:
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l131
https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l274

Looking at the comment on line #130, I saw that you (Luke) suggested they should be runtime commandline args.


I temporarily commented out TRIGGER_ENABLE, while also adding a new flag -d to pass to microwatt-verilator.
https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=066a2da5a68d11fb3d40086b2dcc98fd08a5bd92
https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=cb54a3d858186076ee6b55a5777a63dd76df323d
https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=515738527e751b977a9e6aeb7cc74c967837b94b
https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=59ceebf385cfa2de80ed2b56e8f14a08c28c8d4b

Calling the executable like this will allow trace data to be dumped:
./microwatt-verilator hello_world/hello_world.bin -d


After seeing how big the .vcd file got (about 6GB), I realised that enabling trace dumping by default is a *bad idea*. Thus leaving it behind a flag is probably better. Oh, and do let me know if '-d' is acceptable, can change it to something else.

For the purposes of the tutorial I now have all the features working, so won't proceed with further modification. If you want me to implement some of those other defines as commandline args (TRIGGER_OCCURANCE, TRIGGER_COUNTDOWN, etc., let me know).

Also, I added a -h help flag with a basic help string. It's not much, but at least don't need to reverse engineer the code everytime to know the args.

You can call it like this:
./microwatt-verilator -h

I'll proceed with merging my wiki page into the original, as well as making a tutorial tomorrow.
Comment 16 Luke Kenneth Casson Leighton 2023-05-08 22:27:11 BST
(In reply to Andrey Miroshnikov from comment #15)

> I'll proceed with merging my wiki page into the original, as well as making
> a tutorial tomorrow.

fantastic.
Comment 17 Andrey Miroshnikov 2023-05-09 13:29:19 BST
Merged the new documentation into the old Microwatt page: https://libre-soc.org/HDL_workflow/microwatt/

Will be working on the video next.
Comment 18 Andrey Miroshnikov 2023-05-19 18:09:58 BST
(In reply to Andrey Miroshnikov from comment #17)
> Will be working on the video next.

Video was published on the 12th, on Luke's channel:
https://youtu.be/02LCl3ang8g

Link has also been added to the wiki page.

Closing this one, as now proceeding to ls2.
Comment 19 Andrey Miroshnikov 2023-10-11 15:18:40 BST
This task has been removed from ongoing grant, as the budget is required for other critical tasks. No budget will be assigned to this.
I (Andrey) authorise for this to happen.