Bug 1235 - resolve openpower-isa repository screwups
Summary: resolve openpower-isa repository screwups
Status: IN_PROGRESS
Alias: None
Product: Libre-SOC's first SoC
Classification: Unclassified
Component: Source Code (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- enhancement
Assignee: Luke Kenneth Casson Leighton
URL: https://lists.libre-soc.org/pipermail...
Depends on:
Blocks:
 
Reported: 2023-12-13 11:38 GMT by Luke Kenneth Casson Leighton
Modified: 2023-12-19 02:26 GMT (History)
3 users (show)

See Also:
NLnet milestone: ---
total budget (EUR) for completion of task and all subtasks: 0
budget (EUR) for this task, excluding subtasks' budget: 0
parent task for budget allocation:
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Luke Kenneth Casson Leighton 2023-12-13 11:38:55 GMT
https://bugs.libre-soc.org/show_bug.cgi?id=1228#c26
write-permissions revoked for everyone until this is
resolved. full investigation required.
Comment 1 Luke Kenneth Casson Leighton 2023-12-13 14:52:25 GMT
very quick preliminary investigation, i screwed up "rebase" (agin,
dyslexia) and put some of shriya's WIP into master.
jacob it is critically important you not try to "fix" other
people's screwups.  this is so important it's in the Charter.

https://libre-soc.org/charter/

Everyone is responsible - and accountable - for their actions: good and bad, "Guilt" and "Merit" belong with whoever earned them.

so that "vec" error was for me and shriya to resolve, as an example?
drawing our *attention* to it, not a problem (bugreport). you
actually *fixig* it - Charter violation.
Comment 2 Jacob Lifshay 2023-12-13 18:27:28 GMT
the commit introducing the XLEN access:
https://git.libre-soc.org/?p=openpower-isa.git;a=commitdiff;h=5025354342d8e92f200b9c0c0a04f96a53d7f0e3

author	Luke Kenneth Casson Leighton <lkcl@lkcl.net>	
Tue, 28 Nov 2023 20:41:01 +0000 (20:41 +0000)
committer	Luke Kenneth Casson Leighton <lkcl@lkcl.net>	
Thu, 7 Dec 2023 17:52:56 +0000 (17:52 +0000)

so it was committed only a few days ago.

it looked like a trivial naming mistake to me, hence why I changed it to self.XLEN

whoever was working on that branch that got committed to master by mistake should have been looking at CI for that branch, if they had they should have noticed that there were more failures than expected.

to prevent future mistaken pushes to master, luke maybe install a git hook that requires confirmation before pushing to master?

this makes me think we need to enable CI report emails to everyone in Libre-SOC -- they only get sent to the author and committer of the latest commit on the branch being tested, all branches are independently tested in their own CI runs.
Comment 3 Luke Kenneth Casson Leighton 2023-12-13 23:25:05 GMT
i really don't want hard-coded dependence on a remote CI server
especially one that i (personally) do not have any access to
let alone knowledge of its setup and workings.

(remember: i did instruct you not to use latest-and-greatest,
you disregarded and keep disregarding that, and i just had to
remove *yet another* option from a pytest config file that
terminated the possibility of running pytest under the
*project standard* development environment)

if you can convert the CI environment to be one that uses
*exclusively project standard development environment* and
also allows anyone and i do mean anyone to install it as
a stand-alone setup / VM, then we have a way forward to using
CI.

(i.e. *any* laptop / server used by *any* developer should
 have a devscript that they can personally run and trigger
 unit tests - selectively - to produce results)

right now i do ad-hoc unit tests. i did the best i could on
shriya's laptop (running everything), and obviously missed
something before pushing to the master branch.

if we had a devscript that could install an *entire* standalone
CI then we have a workable solution.
Comment 4 Jacob Lifshay 2023-12-13 23:45:20 GMT
(In reply to Luke Kenneth Casson Leighton from comment #3)
> i really don't want hard-coded dependence on a remote CI server
> especially one that i (personally) do not have any access to
> let alone knowledge of its setup and workings.

other than the access to the fpga, which openpower-isa doesn't use, it is a bog-standard gitlab-runner install using docker. nothing special. the parts that are special are the things that run git mirroring and the program that asks the gitlab server for the build log when it's done, neither of which affect how CI is run.
> 
> (remember: i did instruct you not to use latest-and-greatest,

I'm not. I'm using the version specified in the dev scripts.

I think you probably forgot to update to the latest version of the dev scripts, since they specify the exact same version of pytest used in CI, which does work with that option. the versions were changed a while ago shortly after they were initially added to the dev scripts.

please add the option back in, it substantially improves CI performance iirc.

> if you can convert the CI environment to be one that uses
> *exclusively project standard development environment* and
> also allows anyone and i do mean anyone to install it as
> a stand-alone setup / VM, then we have a way forward to using
> CI.

CI runs in docker by gitlab-runner and is reproducible (unless I missed something, this excludes that it uses the latest master version of the libre-soc repos such as nmutil), all you need is to run it with any gitlab installation.

you can also run it (will be much slower due to lack of build cache) without a whole gitlab install by just installing the gitlab-runner and docker packages and running (in the openpower-isa git repo):
gitlab-runner exec docker build

> (i.e. *any* laptop / server used by *any* developer should
>  have a devscript that they can personally run and trigger
>  unit tests - selectively - to produce results)

there isn't a script, since it's just installing the dev environment and then running pytest.
Comment 5 Jacob Lifshay 2023-12-19 02:26:43 GMT
replying to email:
https://lists.libre-soc.org/pipermail/libre-soc-dev/2023-December/005880.html

On Thu, Dec 14, 2023 at 2:15 AM Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> On Wednesday, December 13, 2023, Jacob Lifshay <programmerjake@gmail.com> wrote:
>
> > please add the option back, it improves scheduling so CI takes substantially less time iirc.

it reduces runtime by 2x -- from 5h45m to 2h37m -- I think we should just enable it by default everywhere and just tell people to use the version of pytest specified in the dev scripts.

> add it on the commandline of the (unofficial, only) CI server.

I added it to .gitlab-ci.yml, which is used by all people trying to run CI on gitlab (just my build server for now afaik, but anyone who wants can easily run their own CI -- just push to a gitlab instance and set the build timeout to like 6hr):
https://git.libre-soc.org/?p=openpower-isa.git;a=commit;h=be18b88c043589b66f8cb6bdd18e9ee274768e76

> https://bugs.libre-soc.org/show_bug.cgi?id=1235#c4
> (TODO: add this bugreport to latest grant proposals...
> done a quick note, cn you remember where the discussion
> on CI for NLnet results was? 1126?)

Here's one spot:
https://bugs.libre-soc.org/show_bug.cgi?id=1126#c34

I don't recall where is the discussion of splitting out only the test cases of interest and couldn't find it after some searching...