Introduction (part of MOU) Developing an ASIC costs millions of dollars but there could be subtle design mistakes. A much lower-cost and faster turn-around is to have a high-performance Hardware-Cycle-Accurate simulator. Normal simulators are too slow to be able to have real-time user interaction. cavatools is therefore specifically designed to give high performance simulation of actual hardware, allowing Hardware Architects to guage whether their Micro-Architecture will meet the end-user requirements, long before money is spent on Mask Charges. ---- (Note: bug #1039 is not part of this grant but was added after 2022-08-071 was signed, instead added to ongoing #961) Task list # Support missing instructions Besides introducing prefixed instructions, SVP64 extensions also introduce many word instructions. Some simply extend vanilla PPC with the new functionality and can be used even outside of SVP64 extensions (for example, a new set of math functions). Others manipulate on SVP64 context and manipulate control flow. The support for all these instructions must also be present in binutils. Budget: 3000 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=958 --------- # Switch reference assembler to machine-readable specs We already have various machine-readable files which describe the instructions, the prefixing rules and other essential important parts. However, the current assembly code does not consider all this information, and instead duplicates some parts of it in the code, or implements some bits of the logic which should have relied on the generation. The current approach leads to code duplication and software rot; what's worse, there's a need to keep track of instructions in the code whenever these are changed in the specifications. This task intends to build the essential parts needed to make most of the assembly to be generated automatically. Budget: 5500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=947 --------- # Support missing specifiers By the time the initial binutils support was implemented, we had no real handling for many of the branch specifiers. It was decided that we should postpone these for a while. The real support for Python assembly was implemented only in September 2022, and has not been merged into binutils since then yet. Budget: 1500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=976 --------- In scope of supporting cavatools, one of crucial pieces is instruction decoder for Power ISA. This task can re-use some of existing infrastructure of the project, relying on nMigen and insndb. It is critically important to provide the code which can be used as a standalone entity: we can and should generate some bits for this code, but it should not rely on external components other than cavatools itself. Budget: 5500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=979 --------- # Implement C-based Power ISA pseudocode compiler Besides the actual decoding to understand which instruction we're going to simulate, we also need a way to specify the actual logic behind the instruction to be executed. We already have human-readable specification in Markdown format, and successfully used it for our reference Python-based simulator. We need to follow the same approach for cavatools, and generate the code for all PowerISA instructions. Budget: 5500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=980 --------- # Support PowerPC ABI in cavatools Besides being able to execute PowerPC instructions in cavatools, we also need to support the PowerPC ABI to execute the real code interacting with the OS. Budget: 4500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=981 --------- # Support PowerPC ABI in ISACaller We need to maintain the reference Python simulator, which also lacks proper userspace ABI support. The ABI support is essential for running real userspace applications. Budget: 4500 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=982 --------- # Support PowerPC SFFS compliance XXXX MUST NOT DUPLICATE WITH bug #999 ! XXXX (resolved by focussing on cavatools having ELF dynamic library loading) We need to continue works on Scalar Fixed-Point + Floating-Point Compliancy Subset (SFFS). The Linux distribution must be compiled and be able to run; this might be a multi-phase work due to the huge amount of packages to support. In particular this stage should focus more on what has to be done to cavatools to get it to run SFFS binaries: ELF linker loading of dynamic libraries. Budget: 6000 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=983 --------- # Support SVP64 in cavatools Once we've reached the stage when we're capable of executing vanilla PowerPC instructions with cavatools, we need to extend this support with SVP64 extensions. Budget: 8000 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=984 --------- # Integrate cavatools into test API We already have a solid infrastructure for testing purposes. However, it does not support cavatools yet. In scope of these tasks, we need to integrate cavatools into our test infrastructure via pygdbmi. Budget: 3000 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=985 --------- # cavatools-related administrative activities Any task includes activities related to issue tracking and monitoring. This includes not only organizing the tasks, but also providing comments to all subtasks, participating in dicussions, budget synchronization and similar activities which rather fit the main task than any of the standalone subtasks. Budget: 3000 EUR URL: https://bugs.libre-soc.org/show_bug.cgi?id=987
system calls in power-gem5: https://github.com/gem5/gem5/blob/stable/src/sim/syscall_emul.cc https://github.com/gem5/gem5/blob/stable/src/arch/power/linux/se_workload.cc#L109 there will be similar in both qemu and cavatools.
qemu system calls: https://github.com/qemu/qemu/blob/master/linux-user/ppc/syscall.tbl here's how they are called: https://github.com/qemu/qemu/blob/master/linux-user/ppc/cpu_loop.c#L336
very interesting! cavatools simply passes the syscall directly to the host operating system, in assembler https://www.felixcloutier.com/x86/syscall.html https://github.com/phaa-eu/cavatools/blob/main/uspike/proxy_syscall.cc#L57
scary! https://stackoverflow.com/questions/37032203/make-syscall-in-python
Well, good news is, I did this a lot, there's nothing new here. :-)
(In reply to Dmitry Selyutin from comment #5) > Well, good news is, I did this a lot, there's nothing new here. :-) i think the python version should do full syscall emulation rather than attempt to proxy to real ones. the only reason proxy works in cavatools is because it is using *actual* userspace memory (!) for ISACaller any "memory" is entirely emulated.
(In reply to Luke Kenneth Casson Leighton from comment #6) > the only reason proxy works in cavatools is because it is using > *actual* userspace memory (!) This sounds really creepy. > for ISACaller any "memory" is entirely emulated. This is the way to go.
imho eur 6000 for the pseudocode to C compiler is fine, don't let our discussion about what exactly we should do for that task block submitting to nlnet.
(In reply to Dmitry Selyutin from comment #7) > (In reply to Luke Kenneth Casson Leighton from comment #6) > > the only reason proxy works in cavatools is because it is using > > *actual* userspace memory (!) > > This sounds really creepy. it's how cavatools gets 250 million instructions per second per process per host core, where other emulators are a minimum of 20x slower and flat-out incapable of multi-core support! peter hsu did not design cavatools to be "pretty and nicey-nicey", he designed it to give developers of applications a good chance at *real-time* high-performance computationally-extreme emulation, so that you can test an application *in full*, iteratiely adding new instructions and new micro-architectures, getting the design right before dropping 16 to 32 ***MILLION*** dollars on Mask Charges, only to find that you utterly screwed up. this is not qemu. we are not in kansas no more. > > for ISACaller any "memory" is entirely emulated. > > This is the way to go. in ISACaller yes. and we can expect performance to degrade even further than it already is. however, we get readable code.
I dropped all the links from the overall task description to keep it reduced to a task list. Any reader should carefully check the entire task discussion anyway, so here these links are: https://libre-soc.org/nlnet_2021_3mdeb_cavatools/ https://github.com/phaa-eu/cavatools https://www.bsc.es/research-and-development/software-and-apps/software-list/cavatools
Updated the milestones for the subtasks. What should be the next step?
Ping to all interested parties.
(In reply to Dmitry Selyutin from comment #11) > Updated the milestones for the subtasks. What should be the next step? i thought it might be sensible to put a "management" budget on here, part for you writing the mou, part for me, and part for future updates to budget-sync and other odds. my feeling is that cavatools eabi is going to be near-trivial, it can come down to EUR 4000. management can be EUR 3000, that leaves EUR 1000 which can go into one or more other tasks which are abit low. #976 and #958?
(In reply to Luke Kenneth Casson Leighton from comment #13) > (In reply to Dmitry Selyutin from comment #11) > > Updated the milestones for the subtasks. What should be the next step? > > i thought it might be sensible to put a "management" budget on here, > part for you writing the mou, part for me, and part for future > updates to budget-sync and other odds. Fair enough, I agree, but with a remark below. > my feeling is that cavatools eabi is going to be near-trivial, > it can come down to EUR 4000. management can be EUR 3000, that > leaves EUR 1000 which can go into one or more other tasks which > are abit low. #976 and #958? I also have a strange feeling that works around integration will need more resources, too. I'm not the one to judge, since I haven't checked this part yet, but I'd be really surprised if integrating cavatools to our testing API would be trivial. But I'm afraid this is to be discovered yet. I'm unlikely to do it due to time constraints, but others will almost certainly handle it. How about the scheme below? 1. Cut ABI tasks to 4500 EUR each (#981, #982). 2. Since we now have 5000 EUR, we can assign 3000 EUR to management, 500 EUR to #976, 500 EUR to #958, and 1000 EUR for wrapup which can migrate to #985 if possible? Or we can assign 1000 EUR #985 immediately if you can confirm this is a difficult one. I'm not sure of the level of gdb support we have now. If OK, I will raise the management task with 3000 EUR, will assign 500 EUR to each #976 and #958, and will move the 1000 EUR to either wrapup or #985, depending on our conclusions.
TL;DR: I feel that #985 is underestimated too; please let me know your opinion.
(In reply to Dmitry Selyutin from comment #14) > Fair enough, I agree, but with a remark below. ack > > my feeling is that cavatools eabi is going to be near-trivial, > > it can come down to EUR 4000. management can be EUR 3000, that > > leaves EUR 1000 which can go into one or more other tasks which > > are abit low. #976 and #958? > > I also have a strange feeling that works around integration will need more > resources, too. I'm not the one to judge, since I haven't checked this part > yet, but I'd be really surprised if integrating cavatools to our testing API > would be trivial. yes, very. cavatools already supports gdbmi - gdb remote machine interface. a new register-set-definition is needed, but that's needed anyway. > But I'm afraid this is to be discovered yet. I'm unlikely > to do it due to time constraints, but others will almost certainly handle > it. How about the scheme below? > > 1. Cut ABI tasks to 4500 EUR each (#981, #982). ahh ISACaller ABI is *not* trivial: unlike cavatools which thunks directly onto existing EABIs with a trivial shim (so as to get absolute max performance) ISACaller has to implement a **FULL** and **TOTAL** implementation... in python. there _may_ be an existing implementation out there but finding it will be a pig: try searching "linux abi implementation in python" > 2. Since we now have 5000 EUR, we can assign 3000 EUR to management, 500 EUR > to #976, 500 EUR to #958, and 1000 EUR for wrapup which can migrate to #985 > if possible? sure. > Or we can assign 1000 EUR #985 immediately if you can confirm this is a > difficult one. I'm not sure of the level of gdb support we have now. full remote gdb machine interface is in cavatools. but only RV. not ppc64le. > If OK, I will raise the management task with 3000 EUR, will assign 500 EUR > to each #976 and #958, and will move the 1000 EUR to either wrapup or #985, > depending on our conclusions.
Updated: https://bugs.libre-soc.org/show_bug.cgi?id=981: 8000 => 4500 https://bugs.libre-soc.org/show_bug.cgi?id=982: 6000 => 4500 https://bugs.libre-soc.org/show_bug.cgi?id=958: 2500 => 3000 https://bugs.libre-soc.org/show_bug.cgi?id=976: 1000 => 1500 https://bugs.libre-soc.org/show_bug.cgi?id=987: 0 => 3000 (management task) https://bugs.libre-soc.org/show_bug.cgi?id=939: 0 => 1000 (this task)
Upd: comment #17
Upd: assigned 1000 from wrapup to cavatools test API support (1000 EUR from #939 => #985)
(In reply to Dmitry Selyutin from comment #17) > https://bugs.libre-soc.org/show_bug.cgi?id=939: 0 => 1000 (this task) really, that EUR 1000 should be in the management task. but you've not put a subtask budget for it. i'm re-running budget-sync to check things. ERROR:root:Total budget assigned to payees (cf_payees_list) doesn't match expected value: bug #987, calculated total 500, expected value 3000 ERROR:root:Bug's assigned milestone doesn't match the milestone assigned to the root bug: descendant bug #988, root bug #961 yep you've assigned EUR 500 to yourself but the remaining EUR 2500 is unassigned *within* that task. what instead must be done is to create a sub-task of #987.
(In reply to Luke Kenneth Casson Leighton from comment #20) > what instead must be done is to create a sub-task of #987. done, sorted. i double-checked the comment #0 totals, they're EUR 50,000. i'll send comment #0 as the Schedule A to Michiel, along with a JSON file.
andrey you added two extra tasks, remember to let people know by putting the commit diff link here https://git.libre-soc.org/?p=libreriscv.git;a=commitdiff;h=a3f5eea083
report on cavatools project. background: luke leighton has been subjected for 16 years to domestic verbal abuse, fleeing for his life from his home on 11 sep 2023. an online article on the chronic damage caused by narcissim is here: https://www.drlamcoaching.com/blog/adrenal-fatigue-and-narcissism/ the article outlines a number of key severely debilitating physical health conditions, the main effects being NEUROLOGICAL damage, and zero tolerance to stress. luke now also has PTSD, in addition to Autism, and also has chronic recurrent glandular fever. the combination of Chronic Adrenal Fatigue and autism is extreme in its consequences, if distressed: IMMEDIATE shock occurs, which combined with the glandular fever (triggered by cortisol, the stress hormone), results in near-instantaneous throat constriction sufficient to make breathing extremely difficult, in some cases requiring hard pressure on his windpipe to temporarily alleviate. summary: stress places luke at risk of DEATH. virtually nobody who has not directly witnessed this phenomena, and even some who have, have any understanding and simply do not believe it is either possible, or in some instances do not believe that the medical diatress is actually occurring, even as luke is desperately struggling to stay alive. there are now over 200 instances of risk of death by choking, many of them video'd and documented. NLnet i regret to report is amongst the people who have not taken this medical phenomena into consideration, despite repeating it well over twenty times to them. they are also aware of the "word" autism but have made no adjustments to their communications protocols to take the risk of medical distress, nor the possibility of death caused by the combination of CFS Autism Glandular Fever and PTSD, into consideration. with that background in mind the report can proceed, bearing in mind that the effects of the Domestic Verbal Abuse by RAYMOND, as well as complete lack of empathy by BHARVANI MARGARITIS CALDERWOOD and LEWIS, had a massive adverse impact on luke's ability to lead the team, resulting in significant communications failures and additinal distress for that team, losing team members and requiring a significant adjustment to the goals achievable. report the original target was to port the c-based program, cavatools, to Power ISA. the amount of money requested turned out to be much smaller than the size of the project, in relation to the skill of available people. the original estimates involved cavatools primary author helping to do the port. unfortunately, the author was unable to help with the instruction decode phase, and the project had to be shelved until someone with the necessary expertise emerged. a second person only emerged with the skill and expertise around october 2023. unfortunately, this person has a significantly higher cost, which meant that it was simply immediately impossible to achieve the original target, because the target had been costed for the original author of cavatools *only*. a meeting was therefore arranged, and NLnet informed, that the project would adapt to focus on the python-based simulator (ISACaller), and on a drastically smaller subset of system calls. unit tests would also be written that could later be used on a subsequent grant, focussed more on the c-based cavatools, bootstrapping it by now checking against known-good unit tests (using the python-based simulator). all of these adapted (expanded) goals, which were part of the original objective and the original application (but not its sole objective) were successfully achieved. interestingly no changes were required to the milestones themselves: the original grant application needed no adaptation: it was just that within the flexibility of the definition of the milestones it was possible to focus more on the python-based simulator. however there is also something very important about the way that the objectives were achieved: as much as possible was *done in c*. a *c* based Power ISA decoder (as a general-purpose library) was written. this meant that a future grant for the c-based cavatools could utilise that c-based general-purpose library. in all, despite the medical difficulties faced by luke (which he was simply not aware of in any way at the time), the grant was definitely successful. summary: sadly, team members were lost along the way due to harrassment and ongoing stalking, and the mental health of everyone on the team seriously adversely affected, but the objectives within the milestones, which did not need changing, were successfully met.
Date: Sat, 20 Jul 2024 01:50:18 +0200 From: Michiel Leenaars <michiel@nlnet.nl> To: Luke Kenneth Casson Leighton <lkcl@lkcl.net> Subject: Re: Re: Get well Dear Luke, (dropping Brian to bcc, this is outside of the scope for him) Thanks for initiating the writeup. This is a step in the right direction. any embarrassment caused by your STATEMENT "you have not achieved the goals" i cannot help you with. you should have listened and ASKED THE QUESTION: "it seems that you have not met the deliverables. have i missed anything here? what actually happened?" Am I correct in interpreting that you suggest you are quoting us saying "you have not achieved the goals"? Where precisely did you get this statement from? We did write: "For each pending payment request, please ensure there is a final comment in the issue tracker indicating where to find the concrete outcome of the task, along with a description and justification of any deviations from the original description. We have attempted to understand your progress, but it is currently opaque and lacks the previous level of structure." What is precisely accusatory in what we wrote? We only ask for clarification, in the clear absence of a wrap up of most invidual tasks and of the project as a whole. What did you find unreasonable in that? Now on the substance: What do you hope to gain by adding your medical and relational history in a public document that is intended as serious proof of work? This isn't about you, for that I suggest establishing a personal blog if need be (in general I would recommend to keep such personal matters outside of the public sphere, as they might block future efforts). This should be a wrap-up of the work from the Libre-SOC team, and as a project lead we expect you to keep the wrap-up professional, factual and on topic. Accusations to e.g. your ex-wife or former colleagues do not belong here, or in fact anywhere on the issue tracker/mailing list/etcetera. Could you please clean this up? The bit under "report" is the only relevant part, but it is not much more than a narrative timeline of how you experienced the project and some decisions made along the way. These may be relevant, but should be added to the individual bugs the payment requests point to instead of this bug that noone will read. This narrative timeline isn't precisely what we would need to wrap up the project though. We need _documentation_ on Libre-SOC.org, explaining e.g. what the c based Power ISA decoder does, where it can be found, how this should (and should not) be used, etc. Very likely you have most of this info somewhere and/or in your head, but the world (including us) just doesn't know where to find it, which prevents us from being able to benefit from your collective work - until you connect the dots. Can you help us by providing such documentation, at a level of professionality matching the significant budget? And can you also make sure that the repo's contain the right information on how to compile, which dependencies are needed, etc? In other words: can you help translate the outcomes to the target audience, in line with the fact that this concerns tax payer money. Best, Michiel
To EC Commission and to Auditors: i am still recovering from abuse and still have a stalker. below you can see that Michiel Lenaars of NLnet is unable to take into consideration the true extent of the damage done to my health. he and Bob Goudriaan have both been informed repeatedly that i have nearly died multiple times as a result of the abuse, but they are unable to appreciate or acknowledge the seriousness. i will attempt to answer inline below, but if it becomes too distressing i will have to stop, recover, and carry on another time: for example, recalling and relating the abusive events when giving statements to the police *caused* medical distress and caused phyisical collapse, every single time. i cannot continue to let that happen: i need to recover from the medical physical effects of the abuse. my recovery is underway but in all seriousness that recovery is very sadly impeded by NLnet not listening. i have attempted to engage Medation services (Brian, Technology Cultivation) but they are not listening to me telling them how distressing that it is that they are not listening, and refused repeatedly to engage Mediation to help get across to them how much harm it is causing to my physical health recovery that they are not listening. i leave it at that and will begin answering below. (In reply to Luke Kenneth Casson Leighton from comment #24) > Date: Sat, 20 Jul 2024 01:50:18 +0200 > From: Michiel Leenaars <michiel@nlnet.nl> > To: Luke Kenneth Casson Leighton <lkcl@lkcl.net> > Subject: Re: Re: Get well > > Dear Luke, > > (dropping Brian to bcc, this is outside of the scope for him) > > Thanks for initiating the writeup. This is a step in the right direction. > > any embarrassment caused by your STATEMENT "you have not > achieved the goals" i cannot help you with. you should > have listened and ASKED THE QUESTION: > > "it seems that you have not met the deliverables. > have i missed anything here? what actually happened?" > > Am I correct in interpreting that you suggest you are quoting us saying "you > have not achieved the goals"? yes. this is literally the first clarifying question that you have ever asked. > Where precisely did you get this statement > from? read bob's (currently private) message again. > We did write: > > "For each pending payment request, please ensure there is a final comment in > the issue tracker indicating where to find the concrete outcome of the task, > along with a description and justification of any deviations from the > original description. We have attempted to understand your progress, but it > is currently opaque and lacks the previous level of structure." that is because i am recovering from deep trauma and shock which has caused organ damage as well as neuro-physiological damage. i have informed you of this repeatedly. > What is precisely accusatory in what we wrote? We only ask for > clarification, in the clear absence of a wrap up of most invidual tasks and > of the project as a whole. What did you find unreasonable in that? you have misunderstood. the failure on your part to listen was well before you asked for this. > Now on the substance: > > What do you hope to gain by adding your medical and relational history in a > public document that is intended as serious proof of work? asking what i "hope to gain" is not an appropriate question, it is shockingly insulting and a distressing accusation that i somehow "benefit". as an autistic adult whom you have been informed many times that i am disabled as a result of that autism, i need you rephrase this question so that it does not start with a distressing accusation. if you are not familiar with how to ask questions of autistic adults so that you do not put them into distress, please look it up or engage the urgent services of Brian (as i have asked you to do for two weeks now) so that i can safely explain it to him, and he can explain it to you. > the wrap-up professional, factual and on topic. Accusations to e.g. your > ex-wife or former colleagues do not belong here, or in fact anywhere on the > issue tracker/mailing list/etcetera. Could you please clean this up? please clarify: 1. are you asking me to censor reports? 2. are you asking me to censor bugreports? 3. are you asking me to censor the mailing list? 4. are you asking me to censor anything else? i cannot continue for today, i am overwhelmed by answering and need to recover so as not to damage my physical and neuro-physiological health.
michiel has just demanded that i censor the discussion, here. the project has always operated on full transparency. for the very first time ever he has demanded that the discussion remain private, which if he had informed me of i would have refused to do. i have declined hs demand to engage in censorship, and am now in considerable distress, even more so than when i replied above. i was with friends, engaging in social discussion as a way to keep endorphin levels up (a necessary part of recovery from abuse) when i read his demand to engage in censorship. it has put me into shock, caused a massive adrenaline spike of fear, which now has me shaking, and i do not know what to do. i am going to lie down now, attempt to sleep in order to recover from the adrenal shock and fear.
nlnet have made it clear that if i do not do what they demand, they will cease and terminate all communication. my autistic disability prohibits me from complying with their demand, even when the financial consequences to me are catastrophic. i do not understand why they have been unable to listen for weeks, now.
one week ago i requested of the Director of NLnet the contact details of the EC and the Auditor. i was told by the Director, words to the effect "i doubt the EC would listen" and the contact details were not given. i have made a second request. i have not received a response.