I've setup a new Libre-SOC wiki page documenting the exact order of instructions I followed to set up a new chroot for working with Microwatt for documentation and reproducibility. Currently stumbling on the same issue I've had back in January https://libre-soc.org/irclog/%23libre-soc.2023-01-25.log.html#t2023-01-25T11:10:47 https://libre-soc.org/irclog/%23libre-soc.2023-01-27.log.html#t2023-01-27T05:30:02 https://libre-soc.org/irclog/%23libre-soc.2023-01-30.log.html#t2023-01-30T10:05:14 -Libre-SOC wiki page: https://libre-soc.org/HDL_workflow/microwatt_tutorial/ The Libre-SOC microwatt tutorial shows the following command to time how long the sim takes to run: (microwatt):$ time ./microwatt-verilator No additional arguments to the binary are specified. As I've covered on the wiki page, the hello_world binary is loaded into RAM at synthesis, so compiled verilator sim should already have the program loaded. In the Makefile, https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;h=610f48d8c89be6d5b9902d7f1bf61f8b6d98ffc0;hb=refs/heads/verilator_trace `RAM_INIT_FILE` (line #144) is specified as `hello world.hex` (the Microwatt 'light bulb' example code). The Makefile uses this `RAM_INIT_FILE` argument for generating the `microwatt.v` verilog file (line #249). Finally, to generate the `microwatt-verilator` binary, `microwatt.v` is pulled in as a dependency (line #254). However, even after running for over 18min on a Ryzen 3600 and verilator set to 11 threads, no terminal output appears. Please help me understand what I'm missing. What would really help is some way to measure the progress of the simulation (and that it's not just sitting, running NOPs). (This bug might be in the wrong category, perhaps documentation is better?)
(In reply to Andrey Miroshnikov from comment #0) > As I've covered on the wiki page, the hello_world binary is loaded into RAM > at synthesis, so compiled verilator sim should already have the program > loaded. mmm.... should be ok. > However, even after running for over 18min on a Ryzen 3600 and verilator set > to 11 threads, no terminal output appears. don't do that arbitrarily. you need to enable the FST compile option which outputs gtkwave traces then go look at them. > What would really help is some way to measure the progress of the simulation > (and that it's not just sitting, running NOPs). gtkwave trace, as i mentioned on the last call.
After our conversation on IRC: https://libre-soc.org/irclog/latest.log.html#t2023-04-27T16:38:23 (In reply to Luke Kenneth Casson Leighton from comment #1) > don't do that arbitrarily. you need to enable the FST compile option > which outputs gtkwave traces then go look at them. > > > What would really help is some way to measure the progress of the simulation > > (and that it's not just sitting, running NOPs). > > gtkwave trace, as i mentioned on the last call. I was able to convert using vcd2fst and open the .fst file in gtkwave. The problem seems to be with the core being idle. Looking at the bram.dump in the hex editor, I see these: pc ff000000 insn ffffffff msr 8000000000000001 pc ff000004 insn ffffffff msr 8000000000000001 pc ff000008 insn ffffffff msr 8000000000000001 rd @ 00000100 di 0 sel ff ........ rd @ 00000101 di 0 sel ff ........ And so on... Looks like PC is starting at 0xff00_0000, which follows the soc.vhdl and microwatt.h definitions for DRAM init. I found the arguments you added for using the BRAM dump or verilator.save files: -s FILENAME Example: ./microwatt-verilator bram.dump Loads a previous dump of the BRAM (which allows you to resume from where you left off) +[TICKNUMBER] Example: ./microwatt-verilator +1999990 Loads from save called "verilator.save.1999990". However it took me a while to finally understand how arguments are passed to verilator itself (using the +verilator prefix). Found the list of runtime verilator args that can be passed: https://verilator.org/guide/latest/exe_sim.html None of those seem relevant. Do I need point the sim to a starting address, or do I need to supply a binary instead? I would've thought that if the microwatt reset address is set to where the code is located, it should begin executing. One more thing I tried looking before finishing for today is looking at the ghdl synthesis in the Makefile. On line #233 and #235 of the Makefile, the (generic?) options -gRAM_INIT_FILE as well as -gRESET_ADDRESS specify the RAM init file to be passed to ghdl, however I haven't found any documention on these options at all.
(In reply to Andrey Miroshnikov from comment #2) > I would've thought that if the microwatt reset address is set to where the > code is located, it should begin executing. indeed - but if you've compiled it to be at a specific memory address, and you "load" it into a different one, where do you think the PC will go on the very first branch?
(In reply to Luke Kenneth Casson Leighton from comment #3) > (In reply to Andrey Miroshnikov from comment #2) > > > I would've thought that if the microwatt reset address is set to where the > > code is located, it should begin executing. > > indeed - but if you've compiled it to be at a specific memory address, > and you "load" it into a different one, where do you think the PC will > go on the very first branch? to the correct spot since nearly all branches are PC-relative? :P
(In reply to Jacob Lifshay from comment #4) > (In reply to Luke Kenneth Casson Leighton from comment #3) > > (In reply to Andrey Miroshnikov from comment #2) > > > > > I would've thought that if the microwatt reset address is set to where the > > > code is located, it should begin executing. > > > > indeed - but if you've compiled it to be at a specific memory address, > > and you "load" it into a different one, where do you think the PC will > > go on the very first branch? > > to the correct spot since nearly all branches are PC-relative? :P generally embedded code needs to know which address it's at, so all the non-pc-relative things work correctly
(In reply to Jacob Lifshay from comment #5) > generally embedded code needs to know which address it's at, so all the > non-pc-relative things work correctly such as "load the immediate address of a bootloader magic constant". you'll find that in the Makefile there is a #define of the offset address, that *absolutely must* be encoded correctly to the address that you expect the binary to be uploaded to. this #define goes into the linker script via some makefile target voodoo involving gcc -E (i think - something like that) which can do macro-substitution.
(In reply to Andrey Miroshnikov from comment #2) > On line #233 and #235 of the Makefile, the (generic?) options > -gRAM_INIT_FILE as well as -gRESET_ADDRESS specify the RAM init file to be > passed to ghdl, however I haven't found any documention on these options at > all. Figured out that "-g" prefixed arguments allow to pass parameters into the VHDL module, so that's where RAM_INIT_FILE, and RESET_ADDRESS arguments go to. Perhaps I should do a basic ghdl counter tutorial, like we have with nmigen, so that devs actually know how to work with this. Even if we don't use VHDL ourselves, still need to know how to access it for Microwatt. (In reply to Luke Kenneth Casson Leighton from comment #6) > such as "load the immediate address of a bootloader magic constant". > > you'll find that in the Makefile there is a #define of the offset > address, that *absolutely must* be encoded correctly to the > address that you expect the binary to be uploaded to. this #define > goes into the linker script via some makefile target voodoo involving > gcc -E (i think - something like that) which can do macro-substitution. Searching for the "bootloader magic constant"... In the Makefile for hello_world, there are no address offset parameters. https://git.libre-soc.org/?p=microwatt.git;a=blob;f=hello_world/Makefile;h=dd51bd38f9ae428328cb8f0d2c3eea479068d997;hb=refs/heads/verilator_trace Makefile provides a RESET_ADDRESS of 65280 (0xff00), because VHDL doesn't allow passing integers bigger than 2^32 apparently. Then in top-generic.vhdl: https://git.libre-soc.org/?p=microwatt.git;a=blob;f=fpga/top-generic.vhdl;h=0ce8456d0f781e858a4d70c8ffe46b90c119d216;hb=refs/heads/verilator_trace#l108 This integer is converted to an unsigned 48-bit number, and concatinated with lower 16-bits. This gives the correct reset address of 0xff00_0000. The RAM_INIT_FILE (in my case the hello_world binary) is passed to the soc.vhdl, and then to wishbone_bram_wrapper.vhdl. https://git.libre-soc.org/?p=microwatt.git;a=blob;f=soc.vhdl;h=b02fe12c6b9d2120c759c6bb3886bb8d6ad917ee;hb=refs/heads/verilator_trace https://git.libre-soc.org/?p=microwatt.git;a=blob;f=wishbone_bram_wrapper.vhdl;h=4e3fb506dfecb66bb6bf165251957b5d2547f26d;hb=refs/heads/verilator_trace Inside wishbone_bram_wrapper.vhdl, the main_bram module (with RAM_INIT_FILE) is only generated if the SIM_MAIN_BRAM parameter is true. The Makefile sets it to false by default, which would explain with the RAM is not actually initialised. Setting SIM_MAIN_BRAM to true in the Makefile causes an error during "make microwatt-verilator": ERROR: vhdl import failed. make: *** [Makefile:252: microwatt.v] Error 1 Earlier, this error occurs about multiple assignments in wishbone_bram_wrapper.vhdl wishbone_bram_wrapper.vhdl:23:9:error: multiple assignments for "wishbone_out" offsets 0:63 wishbone_out : out wishbone_slave_out; I'm guessing I don't need this SIM_MAIN_BRAM parameter, why else would it be disabled. Another parameter I tried to change (after reverting SIM_MAIN_BRAM to false) is SIM_BRAM_CHAINBOOT. In the Makefile it is set to 6291456 (0x600000, needed for the linux image). Setting it to 0, I get the following error: error: override for generic "sim_bram_chainboot" is out of bounds ERROR: vhdl import failed. make: *** [Makefile:251: microwatt.v] Error 1 In soc.vhdl line #62, the default value is set to 0, so I don't know why it complains about being out of bounds. Other than this, I have not found any other offset values to adjust in the Makefile for microwatt-verilator. microwatt_soc.h (used for compiling PowerISA binaries to run) has some potentially relevant parameters: https://git.libre-soc.org/?p=microwatt.git;a=blob;f=include/microwatt_soc.h;h=989721b14e57781c3099b10936e6f91c7b69c767;hb=refs/heads/verilator_trace MEMORY_BASE=0x0000_0000 /* "Main" memory alias, either BRAM or DRAM */ DRAM_BASE=0x4000_0000 /* DRAM if present */ BRAM_BASE=0x8000_0000 /* Internal BRAM */ Also there's an option if STANDALONE_MINI_BIOS is defined: DRAM_INIT_BASE=0x0000_0000 /* alternative, for verilator simulation */ Otherwise DRAM_INIT_BASE=0xff00_0000 /* Internal DRAM init firmware */ Also, after looking at IRC logs: https://libre-soc.org/irclog/%23libre-soc.2022-01-26.log.html#t2022-01-26T19:24:55 Saw that a binary can be passed in to microwatt-verilator: ./microwatt-verilator tests/hello_world/hello_world.bin However I'm still missing the correct offset setting, or something like that. The bram.dump first few lines: pc ff000000 insn ffffffff msr 8000000000000001 pc ff000004 insn ffffffff msr 8000000000000001 pc ff000008 insn ffffffff msr 8000000000000001 rd @ 00000100 di 48000000 sel ff ...H.... rd @ 00000101 di 0 sel ff ........ Not sure where to go from here. Is there a way to search for a string of text in all the email and IRC archives? Your previous updates from 2021-2022 on Microwatt would be handy, and I'd prefer not to go through months of IRC logs and emails by hand, if there's a way to search it.
ok you've misunderstood: the *executable binary* you have compiled with an incorrect offset. check the linker script.
(In reply to Luke Kenneth Casson Leighton from comment #8) > ok you've misunderstood: the *executable binary* you have compiled with > an incorrect offset. check the linker script. https://git.libre-soc.org/?p=microwatt.git;a=blob;f=hello_world/powerpc.lds;h=96bc6b9b16120335e73f6e90c29b3573d2c55e59;hb=refs/heads/verilator_trace I'm assuming "_start = .;" is the entry point, and the address is equal to the first available (0x0). Using objdump, viewed the regions of the elf file before changing anything. objdump -x hello_world.elf <- shows all headers objdump -h hello_world.elf <- shows section headers -h shows the section list (only first three shown): Sections: Idx Name Size VMA LMA File off Algn 0 .head 00000f84 0000000000000000 0000000000000000 00010000 2**0 CONTENTS, ALLOC, LOAD, READONLY, CODE 1 .sfpr 00000014 0000000000001000 0000000000001000 00011000 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 2 .text.startup.main 00000070 0000000000001014 0000000000001014 00011014 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE -x showed the symbol table, and the main code seems to be where .text.startup.main is: 0000000000000000 l d .head 0000000000000000 .head 0000000000001000 l d .sfpr 0000000000000000 .sfpr 0000000000001014 l d .text.startup.main 0000000000000000 .text.startup.main ... 0000000000000000 g .head 0000000000000000 _start 0000000000001014 g F .text.startup.main 0000000000000070 0x60 main which is why I tried setting the _start to 0x1014. Tried changing _start to: - 0x1000 (where .text section begins) - 0xff00_1000 (my guess what the SoC level address would be) - 0x1014 - 0xff00_1014 The last one I tried (_start = 0xff001014;) changed the bram.dump slightly, however as before (and what I didn't show from previous bram.dump, sorry) is that the PC is stuck at 0x800: pc ff000000 insn ffffffff msr 8000000000000001 pc ff000004 insn ffffffff msr 8000000000000001 pc ff000008 insn ffffffff msr 8000000000000001 rd @ 00000100 di 48000000 sel ff ...H.... rd @ 00000101 di 0 sel ff ........ rd @ 00000102 di 0 sel ff ........ rd @ 00000103 di 0 sel ff ........ pc 800 insn 48000000 msr 8000000000000001 rd @ 00000104 di 0 sel ff ........ rd @ 00000105 di 0 sel ff ........ rd @ 00000106 di 0 sel ff ........ pc 800 insn 48000000 msr 8000000000000001 rd @ 00000107 di 0 sel ff ........ pc 800 insn 48000000 msr 8000000000000001 I'm not really sure how the PC goes back from ff00_0008 to 0x800.
(In reply to Andrey Miroshnikov from comment #9) > I'm not really sure how the PC goes back from ff00_0008 to 0x800. an exception. look up the table somewhere... page 1067 rings bell somewhy.
(In reply to Luke Kenneth Casson Leighton from comment #10) > an exception. look up the table somewhere... page 1067 rings > bell somewhy. Found the table you mentioned: PowerISA v3.1B pdf: Section 7.5 Interrupt Definitions PDF page 1302 (spec page 1276) Figure 70. Effective address of interrupt vector by interrupt type: ... 00..0000_0800 | Floating-Point Unavailable ... So I'm guessing this is something about floating-point hardware not being present? Looking at the bram.dump line in detail: pc ff000000 insn ffffffff msr 8000000000000001 Does it mean: PC: ff00_0000 decoded instruction: ffff_ffff (I'm guessing this is incorrect decoding, because the word is not an instruction but data, or empty) MSR value: 8000000000000001 Machine State Register (Book III, Section 4.2.1): bit0: 0 (32-bit mode) / 1 (64-bit mode) ... (ignoring the rest) bit63: 0 (big-endian) / 1 (little-endian) So just out of reset the processor is: - in 64-bit thread mode, little-endian - PC at ff00_0000 - Current instruction is ffff_ffff (not a valid instruction?) It looks like (as you have pointed out already), the processor is expecting instructions at ff00_0000 (reset address), but the program binary is not there. It goes through addresses ff00_0000, ff00_0004, ff00_0008, but continues to find no instructions. What do these statements mean: rd @ 00000100 di 48000000 sel ff ...H.... Read value at address 0000_0100 is 4800_0000? Sorry for being dumb, I think I understand the problem, but I don't know how to go about pointing the cpu to the right place.
(In reply to Andrey Miroshnikov from comment #11) > Sorry for being dumb, I think I understand the problem, but I don't know how > to go about pointing the cpu to the right place. i already said: you don't. you compile the *binary* to be at the correct place and place the *binary* at that memory location by specifying the correct options to ls2 to place the *binary* at that address on the wishbone bus.
lkcl@fizzy:~/src/libresoc/microwatt2$ git diff diff --git a/Makefile b/Makefile index 610f48d..17d1548 100644 --- a/Makefile +++ b/Makefile @@ -214,10 +214,10 @@ _fpga_files = fpga/soc_reset.vhdl \ nonrandom.vhdl # use an alternative core (in verilog) -EXTERNAL_CORE=false +EXTERNAL_CORE=true # VHDL does not allow integers greater than 2^32, so shift down # by 16 bits and add 16 bits zeros back on in soc-generic.vhdl -RESET_ADDRESS=65280 # 0xff00_0000>>16 +RESET_ADDRESS=0 # 65280 # 0xff00_0000>>16
IRC logs from yesterday: https://libre-soc.org/irclog/latest.log.html#t2023-05-01T15:07:37 (In reply to Luke Kenneth Casson Leighton from comment #13) > -RESET_ADDRESS=65280 # 0xff00_0000>>16 > +RESET_ADDRESS=0 # 65280 # 0xff00_0000>>16 As Luke has suggested: https://libre-soc.org/irclog/latest.log.html#t2023-05-02T00:30:56 Thanks to clearing up what I was actually doing (running Microwatt by itself and *not* in the ls2 fabric), and going ahead with changing the RESET_ADDRESS parameter to 0 in: https://git.libre-soc.org/?p=microwatt.git;a=blob;f=Makefile;h=610f48d8c89be6d5b9902d7f1bf61f8b6d98ffc0;hb=refs/heads/verilator_trace#l220 fixed the problem with verilator, and hello_world ran. In the IRC chat Luke also mentioned a makefile to automatically generate the correct powerpc.lds with the appropriate starting address, found in ls2 repo: https://git.libre-soc.org/?p=ls2.git;a=blob;f=hello_world/Makefile;h=50f039112f54165f8f6f7421ac62be1661889576;hb=HEAD#l9 Later today, I will test this with Microwatt to make sure the original RESET_ADDRESS 0xff00_0000 works, update the wiki tutorial, and see about recording a short video to demonstrate setting up, and running code.
After getting the sim to run last week, I forgot that the vcd trace file was empty. Luke helped out on IRC: https://libre-soc.org/irclog/%23libre-soc.2023-05-07.log.html#t2023-05-07T14:42:12 I discovered the reason: the tick() function has a boolean 'dump' argument which, if set to true, will dump the trace data for the current clock cycle. The reason it was dumping the first time 0, is because during reset condition, tick() dump argument is hard-coded to true: https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l261 The reason there was no further dumping, is because the TRIGGER_ENABLE define was causing the 'traceme' dump flag to be set to false: https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l131 https://git.libre-soc.org/?p=microwatt.git;a=blob;f=verilator/microwatt-verilator.cpp;h=a226393f6ba74d5e3e1ffdb729d731d2311d53ad;hb=f106b4a3ab6859c2ab54e8377609e643a4eef1e6#l274 Looking at the comment on line #130, I saw that you (Luke) suggested they should be runtime commandline args. I temporarily commented out TRIGGER_ENABLE, while also adding a new flag -d to pass to microwatt-verilator. https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=066a2da5a68d11fb3d40086b2dcc98fd08a5bd92 https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=cb54a3d858186076ee6b55a5777a63dd76df323d https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=515738527e751b977a9e6aeb7cc74c967837b94b https://git.libre-soc.org/?p=microwatt.git;a=commitdiff;h=59ceebf385cfa2de80ed2b56e8f14a08c28c8d4b Calling the executable like this will allow trace data to be dumped: ./microwatt-verilator hello_world/hello_world.bin -d After seeing how big the .vcd file got (about 6GB), I realised that enabling trace dumping by default is a *bad idea*. Thus leaving it behind a flag is probably better. Oh, and do let me know if '-d' is acceptable, can change it to something else. For the purposes of the tutorial I now have all the features working, so won't proceed with further modification. If you want me to implement some of those other defines as commandline args (TRIGGER_OCCURANCE, TRIGGER_COUNTDOWN, etc., let me know). Also, I added a -h help flag with a basic help string. It's not much, but at least don't need to reverse engineer the code everytime to know the args. You can call it like this: ./microwatt-verilator -h I'll proceed with merging my wiki page into the original, as well as making a tutorial tomorrow.
(In reply to Andrey Miroshnikov from comment #15) > I'll proceed with merging my wiki page into the original, as well as making > a tutorial tomorrow. fantastic.
Merged the new documentation into the old Microwatt page: https://libre-soc.org/HDL_workflow/microwatt/ Will be working on the video next.
(In reply to Andrey Miroshnikov from comment #17) > Will be working on the video next. Video was published on the 12th, on Luke's channel: https://youtu.be/02LCl3ang8g Link has also been added to the wiki page. Closing this one, as now proceeding to ls2.
This task has been removed from ongoing grant, as the budget is required for other critical tasks. No budget will be assigned to this. I (Andrey) authorise for this to happen.