Bug 714 - Coriolis2 Installation as Normal User Script
Summary: Coriolis2 Installation as Normal User Script
Status: RESOLVED FIXED
Alias: None
Product: Libre-SOC's first SoC
Classification: Unclassified
Component: Source Code (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- normal
Assignee: Luke Kenneth Casson Leighton
URL: https://libre-soc.org/HDL_workflow/co...
Depends on:
Blocks: 384
  Show dependency treegraph
 
Reported: 2021-10-01 19:48 BST by Andrey Miroshnikov
Modified: 2021-12-09 14:25 GMT (History)
2 users (show)

See Also:
NLnet milestone: NLNet.2019.10.Wishbone
total budget (EUR) for completion of task and all subtasks: 350
budget (EUR) for this task, excluding subtasks' budget: 350
parent task for budget allocation: 384
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:
[andrey] amount = 350 submitted = 2021-10-31 paid = 2021-11-17


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Andrey Miroshnikov 2021-10-01 19:48:55 BST
This bug is for a task given to me (andrey) by lkcl to split the existing "coriolis2-chroot" script's functionality:
-Schroot creation (delegated to "mk-deb-chroot")
-Installation of dependencies, yosys, coriolis, alliance, alliance-check-toolkit, as a *normal user*

The original script can be found in "dev-env-setup" repo.

The following strategy was suggested: 
1) take a copy [of "coriolis2-chroot"]
2) remove the parts that set up an schroot [now done by "mk-deb-chroot"]
3) convert anything that requires to be run as a user to just... run as a user
4) convert anything that needs root (make install) to use sudo
5) convert the *existing* coriolis2-chroot to:
  a) call mk-deb-chroot
  b) call the new script created from 1-4 running it
       *in* the newly-created chroot.
       this by actually copying the script over to it

After fixing the Alliance installation problem (documented in bug #696), I'm carrying on with this task.


Scripts that exist so far:
-To run as root (sudo bash) in the host environment:
"mk-deb-chroot" [WORKING]: Create a schroot environment. Already existed.
"cp-scripts-to-chroot" [WORKING]: Copy the "dev-env-setup" directory to given chroot (by default you chroot is not able to access the libre-soc git repos unless you've copied the SSH keys over).
"rm-deb-chroot" [WORKING]: After doing several tests, thought it'd be better to have a cleanup script if you no longer need a schroot environment. Modifies your hosts schroot.conf!

-To run as normal user in the chroot environment:
"coriolis-install" [IN PROGRESS]: Script to be run by a user once inside your chroot. For the moment still working on.
Comment 1 Andrey Miroshnikov 2021-10-01 20:47:30 BST
Noticed the reason why "coriolisEnv.py" fails when executed from shell script.

The python script calls the unix utility "which" with an argument of the shell.
The command looks like this:
whichCommand  = subprocess.Popen ( ['which', shell ], stdout=subprocess.PIPE )

Where "shell" is determined by:
shell         = psCommand.stdout.readlines()[1][:-1].split()[3].lstrip('-')

When ran manually, "shell" returned is:
Shell: bash
which is what we want.

When ran inside a shell script, the name of the script is returned instead! At the moment I'm using a test script called "test-coriolisenv" and the value returned is the first 15 chars of the script name:
Shell: test-coriolisen


The method stdout.readline() of "whichCommand" is correct when ran manually (the path to bash):
['/usr/bin/bash\n']
And empty when inside the script (because there's no programme called "test-coriolisen" hahaha!)
[]


I'm happy at least that I identified the issue, now I need to think how to fix it. Certainly renaming "coriolis-install" to "bash" wouldn't be very prudent. I'll look further into it tomorrow.
Comment 2 Luke Kenneth Casson Leighton 2021-10-01 20:51:32 BST
(In reply to andrey from comment #1)
> Noticed the reason why "coriolisEnv.py" fails when executed from shell
> script.
> 
> The python script calls the unix utility "which" with an argument of the
> shell.

and we didn't encounter this before because the chroot did this:

chroot /bin/bash << EOF
coriolisEnv.py > $HOME/coriolisenv
EOF

which *is* running /bin/bash.  whoops.
Comment 3 Luke Kenneth Casson Leighton 2021-10-01 20:54:23 BST
https://gitlab.lip6.fr/vlsi-eda/coriolis/-/issues/48

andrey a workaround is to do "/bin/bash -c coriolisEnv.py >" blah blah

or use the EOF trick again from comment 2:

/bin/bash << EOF
.... coriolisEnv.py > blahblah
EOF
Comment 4 Luke Kenneth Casson Leighton 2021-10-01 21:00:38 BST
Andrey, this one we can put as a "documentation" budget,
if you can update the tutorials and pages involved that
explain how to use these two scripts.

also, Jean-Paul will be doing a coriolis_install_python3 which
should also be mentioned.
Comment 5 Jean-Paul.Chaput 2021-10-01 21:44:41 BST
(In reply to andrey from comment #1)
> Noticed the reason why "coriolisEnv.py" fails when executed from shell
> script.
> 
> I'm happy at least that I identified the issue, now I need to think how to
> fix it. Certainly renaming "coriolis-install" to "bash" wouldn't be very
> prudent. I'll look further into it tomorrow.

  Whoops. Didn't foresee this use-case. The problem to solve here
  is to reliably find the shell the script is running under so it
  can select the right syntax to issue environment commands.
  For various Linux/UNIX and MacOS...

  Maybe the answer is "we cannot" and we should force the user to
  supply it as an argument to the coriolisEnv.py script.
Comment 6 Luke Kenneth Casson Leighton 2021-10-01 21:54:20 BST
(In reply to Jean-Paul.Chaput from comment #5)

>   Maybe the answer is "we cannot" and we should force the user to
>   supply it as an argument to the coriolisEnv.py script.

https://unix.stackexchange.com/questions/352316/finding-out-the-default-shell-of-a-user-within-a-shell-script

$SHELL.
Comment 7 Jean-Paul.Chaput 2021-10-01 22:08:21 BST
(In reply to Luke Kenneth Casson Leighton from comment #6)
> (In reply to Jean-Paul.Chaput from comment #5)
> 
> >   Maybe the answer is "we cannot" and we should force the user to
> >   supply it as an argument to the coriolisEnv.py script.
> 
> https://unix.stackexchange.com/questions/352316/finding-out-the-default-
> shell-of-a-user-within-a-shell-script
> 
> $SHELL.

Of course that was my first idea... But, what if you do that,
assuming that your login shell is /bin/bash :

bash> echo $SHELL
/bin/bash
bash> tcsh
tcsh> echo $SHELL
/bin/bash

And this is a real use case. As users usually are under bash,
*but* VLSI CAD tools very much like C-Shell.
Comment 8 Luke Kenneth Casson Leighton 2021-10-01 22:29:38 BST
(In reply to Jean-Paul.Chaput from comment #7)

> And this is a real use case. As users usually are under bash,
> *but* VLSI CAD tools very much like C-Shell.

ok so it's the shell you are running *at the time* that is needed,
not the user's login shell.

which they don't change in /etc/passwd.

sigh.
Comment 9 Luke Kenneth Casson Leighton 2021-10-01 22:31:42 BST
https://www.google.com/search?q=reliably+get+current+shell+linu

Use the following Linux or Unix commands:
ps -p $$ – Display your current shell name reliably.

apparently.
Comment 10 Jean-Paul.Chaput 2021-10-01 22:40:34 BST
(In reply to Luke Kenneth Casson Leighton from comment #9)
> https://www.google.com/search?q=reliably+get+current+shell+linu
> 
> Use the following Linux or Unix commands:
> ps -p $$ – Display your current shell name reliably.
> 
> apparently.

  Ha ha. We are going back to what coriolisEnv.py is doing (although
  it is a little bit more contrived).

  This command, if run inside a shell script, will return the
  name of the script, not the shell that execute it (just tried).
Comment 11 Luke Kenneth Casson Leighton 2021-10-01 23:21:42 BST
(In reply to Jean-Paul.Chaput from comment #10)

>   Ha ha. We are going back to what coriolisEnv.py is doing (although
>   it is a little bit more contrived).
> 
>   This command, if run inside a shell script, will return the
>   name of the script, not the shell that execute it (just tried).

ooooooOOO so annoying.

sounds like "coriolisEnv.py --shell `ps -p $$`" then.
Comment 12 Luke Kenneth Casson Leighton 2021-10-01 23:22:50 BST
or, hunt back from parent process parent process parent process until
hitting a shell.
Comment 13 Jean-Paul.Chaput 2021-10-01 23:58:03 BST
(In reply to Luke Kenneth Casson Leighton from comment #12)
> or, hunt back from parent process parent process parent process until
> hitting a shell.

  Yes, that was also my conclusion *if* we want to keep the
  auto-detection... Sight!
Comment 14 Andrey Miroshnikov 2021-10-02 08:47:50 BST
(In reply to Luke Kenneth Casson Leighton from comment #3)
> https://gitlab.lip6.fr/vlsi-eda/coriolis/-/issues/48
> 
> andrey a workaround is to do "/bin/bash -c coriolisEnv.py >" blah blah
> 
> or use the EOF trick again from comment 2:
> 
> /bin/bash << EOF
> .... coriolisEnv.py > blahblah
> EOF
Good idea, I'll try both of those.

In my test script had the following (modified coriolisEnv.py to only print the shell):
# First method
/bin/bash -c /home/$USER/coriolis-2.x/src/coriolis/bootstrap/coriolisEnv.py
# Second method
/bin/bash << EOF
/home/$USER/coriolis-2.x/src/coriolis/bootstrap/coriolisEnv.py
EOF

The standard output:
Test script for checking coriolisEnv.py!
Shell: test-coriolisen
whichCommand.stdout.readlines(): []
Shell: bash
whichCommand.stdout.readlines(): ['/usr/bin/bash\n']

Section on "-c" in the bash man pages:
"If the -c option is present, then commands are read from  the
first non-option argument command_string.  If there are argu-
ments after the command_string, the  first  argument  is  as-
signed  to $0 and any remaining arguments are assigned to the
positional parameters.  The assignment to $0 sets the name of
the shell, which is used in warning and error messages."

Is "/bin/bash -c" not going to work because of $0? I tried adding "bash" after the python script (/bin/bash -c .../coriolisEnv.py bash) to see if that would set $0, but didn't work.

So the first method doesn't work, however the second does, so I'll stick to the second one for now.



(In reply to Luke Kenneth Casson Leighton from comment #4)
> Andrey, this one we can put as a "documentation" budget,
> if you can update the tutorials and pages involved that
> explain how to use these two scripts.
Sure. Is it just the coriolis2 wiki page I need to work on?
https://libre-soc.org/HDL_workflow/coriolis2/

There's also the devscripts wiki:
https://libre-soc.org/HDL_workflow/devscripts/
where I can include brief descriptions of the other scripts. Is that worth doing?

> also, Jean-Paul will be doing a coriolis_install_python3 which
> should also be mentioned.
Sure, I'll mention what needs to be updated to work with python3. Now that we found the causes of my issues, a python3 version *might* work, so I'll create a second script called "coriolis-install-py3" (with a disclaimer saying it probably doesn't work yet) after I'm done with "coriolis-install" and updated the documentation.


(In reply to Jean-Paul.Chaput from comment #13)
> (In reply to Luke Kenneth Casson Leighton from comment #12)
> > or, hunt back from parent process parent process parent process until
> > hitting a shell.
> 
>   Yes, that was also my conclusion *if* we want to keep the
>   auto-detection... Sight!
Well, at least we can use the trick Luke mentioned in comment #3 for now.
Comment 15 Andrey Miroshnikov 2021-10-02 08:52:13 BST
(In reply to Luke Kenneth Casson Leighton from comment #11)
> >   This command, if run inside a shell script, will return the
> >   name of the script, not the shell that execute it (just tried).
> 
> ooooooOOO so annoying.
> 
> sounds like "coriolisEnv.py --shell `ps -p $$`" then.
Also tried:
.../coriolisEnv.py --shell `ps -p $$`
didn't work either (also returned name of the script).
Comment 16 Luke Kenneth Casson Leighton 2021-10-02 10:22:01 BST
(In reply to andrey from comment #15)

> Also tried:
> .../coriolisEnv.py --shell `ps -p $$`
> didn't work either (also returned name of the script).

it won't - that was a prototype idea. coriolisEnv.py does not take
an option "--shell".

(In reply to andrey from comment #14)

> (In reply to Luke Kenneth Casson Leighton from comment #4)

> Sure. Is it just the coriolis2 wiki page I need to work on?
> https://libre-soc.org/HDL_workflow/coriolis2/

yes.
 
> There's also the devscripts wiki:
> https://libre-soc.org/HDL_workflow/devscripts/
> where I can include brief descriptions of the other scripts. Is that worth
> doing?

hmmm, that one i think should be kept very brief.
Comment 17 Luke Kenneth Casson Leighton 2021-10-02 11:17:27 BST
andrey, why is there an identical script, one with a minus symbol
and one with an underscore?

lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u coriolis-install coriolis_install 
lkcl@fizzy:~/src/libresoc/dev-env-setup$

(and also an identical one cp-scripts-to-chroot and .sh)

lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u cp-scripts-to-chroot cp-scripts-to-chroot.sh 
lkcl@fizzy:~/src/libresoc/dev-env-setup$
Comment 18 Andrey Miroshnikov 2021-10-02 14:09:24 BST
(In reply to Luke Kenneth Casson Leighton from comment #17)
> andrey, why is there an identical script, one with a minus symbol
> and one with an underscore?
> 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u coriolis-install
> coriolis_install 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$
> 
> (and also an identical one cp-scripts-to-chroot and .sh)
> 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u cp-scripts-to-chroot
> cp-scripts-to-chroot.sh 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$

The original script was "coriolis_install" with an underscore (my mistake), and I used mv to change the name (to better fit the rest of the scripts). Same goes for "cp-scripts-to-chroot.sh" where I removed the .sh extension.

Unfortunately git only deleted the old scripts in my local files, not on the repo. When I do a git pull, the old versions don't get downloaded. Should I use rm next time instead of mv?
Comment 19 Andrey Miroshnikov 2021-10-02 16:51:34 BST
(In reply to andrey from comment #18)
> (In reply to Luke Kenneth Casson Leighton from comment #17)
> > andrey, why is there an identical script, one with a minus symbol
> > and one with an underscore?
> > 
> > (and also an identical one cp-scripts-to-chroot and .sh)
> 
> Unfortunately git only deleted the old scripts in my local files, not on the
> repo. When I do a git pull, the old versions don't get downloaded.

Had to reset my local repo to the latest with
git reset --hard HEAD
and the duplicate files appeared, which I now removed. Checked web view of git and old files no longer there.
Comment 20 Luke Kenneth Casson Leighton 2021-10-02 16:56:57 BST
(In reply to andrey from comment #18)

> The original script was "coriolis_install" with an underscore (my mistake),
> and I used mv to change the name 

you should use "git mv" for that situation.

there is a fundamental difference between the git repository
and the "index" which is the checked-out contents that happen
to be on *your* local filesystem.

the two should not be confused or conflated.
Comment 21 Luke Kenneth Casson Leighton 2021-10-02 17:00:50 BST
(In reply to andrey from comment #18)

> Unfortunately git only deleted the old scripts in my local files, not on the
> repo. When I do a git pull, the old versions don't get downloaded. Should I
> use rm next time instead of mv?

the local command that is part of bash known as "rm"
would simply remove a local file.

running the local command "rm" would do absolutely nothing to the
git epository.

if however you meant *GIT RM* then obviously that is completely
different from *GIT MV*.  i leave it to you to look up the man
page on git rm and git mv?
Comment 22 Jean-Paul.Chaput 2021-10-04 10:10:31 BST
(In reply to Luke Kenneth Casson Leighton from comment #16)
> (In reply to andrey from comment #15)
> 
> > Also tried:
> > .../coriolisEnv.py --shell `ps -p $$`
> > didn't work either (also returned name of the script).
> 
> it won't - that was a prototype idea. coriolisEnv.py does not take
> an option "--shell".

  The latest version (Python 3) now has it.
Comment 23 Andrey Miroshnikov 2021-10-07 12:18:42 BST
I've made a top-level script that sets up the chroot and Coriolis2 (as outlined in comment #1). Ran it once (after making sure "coriolis-install" already worked), and it works (from creating chroot all the way to displaying the chip_r floorplan)!


The default chroot name I'm using is "coriolis". Should I change this, or perhaps provide an optional input argument?


The script also checks if a chroot exists, and if it does, exits before doing anything. This is because I don't expect a user to be re-running the script more than once.
Once the chroot is setup, "coriolis-install" can be run independently.

The wiki pages for Coriolis2 and devscripts have been updated (I moved the chroot information there as it is applicable to other software). Have a go running the script "coriolis2-chroot" if you have time (probably before going off for lunch or something).


Also after checking the color codes used in mk-deb-chroot, I found a convenient way to change echo text colour:
RED='\033[1;91m'
NC='\033[0m'
echo -e "${RED}This text is in red,${NC} while this text is plain (no colour)."

There are even more options to look at:
http://www.cplusplus.com/forum/unices/36461/
https://stackoverflow.com/questions/5947742/how-to-change-the-output-color-of-echo-in-linux

"coriolis2-chroot" uses these colour variables for some of the messages.
Comment 24 Luke Kenneth Casson Leighton 2021-10-07 12:50:08 BST
(In reply to andrey from comment #23)
> I've made a top-level script that sets up the chroot and Coriolis2 (as
> outlined in comment #1). Ran it once (after making sure "coriolis-install"
> already worked), and it works (from creating chroot all the way to
> displaying the chip_r floorplan)!

hooraaay :)

> 
> The default chroot name I'm using is "coriolis". Should I change this, or
> perhaps provide an optional input argument?

ah yes that's a good idea
 
> 
> The script also checks if a chroot exists, and if it does, exits before
> doing anything. This is because I don't expect a user to be re-running the
> script more than once.
> Once the chroot is setup, "coriolis-install" can be run independently.

good idea
 
> The wiki pages for Coriolis2 and devscripts have been updated (I moved the
> chroot information there as it is applicable to other software).

i saw that.  nice touch about referring to documentation in the script.
i'm glad you kept the original "manual" instructions, it helps for
people who want to know what is going on.

>  Have a go
> running the script "coriolis2-chroot" if you have time (probably before
> going off for lunch or something).

well, i already have a chroot named coriolis (and use it) so that would
clearly be bad :) and would justify the extra argument.

> 
> 
> Also after checking the color codes used in mk-deb-chroot, I found a
> convenient way to change echo text colour:
> RED='\033[1;91m'
> NC='\033[0m'
> echo -e "${RED}This text is in red,${NC} while this text is plain (no
> colour)."

nice.
Comment 25 Andrey Miroshnikov 2021-10-07 13:49:46 BST
(In reply to Luke Kenneth Casson Leighton from comment #24)
> hooraaay :)
Indeed, it's nice to see fruits of your labour start to bear fruit!

> > The default chroot name I'm using is "coriolis". Should I change this, or
> > perhaps provide an optional input argument?
> 
> ah yes that's a good idea
Implemented.
 
> > The wiki pages for Coriolis2 and devscripts have been updated (I moved the
> > chroot information there as it is applicable to other software).
> 
> i saw that.  nice touch about referring to documentation in the script.
> i'm glad you kept the original "manual" instructions, it helps for
> people who want to know what is going on.
Documentation's already there, may as well make it as accessible as possible ;)

> >  Have a go
> > running the script "coriolis2-chroot" if you have time (probably before
> > going off for lunch or something).
> 
> well, i already have a chroot named coriolis (and use it) so that would
> clearly be bad :) and would justify the extra argument.
Now should be possible, I tested it the optional argument with just setting up the chroot.


Is there anything else needed for this task, or is it complete?
Comment 26 Luke Kenneth Casson Leighton 2021-10-07 13:56:51 BST
(In reply to andrey from comment #25)
> (In reply to Luke Kenneth Casson Leighton from comment #24)
> > hooraaay :)
> Indeed, it's nice to see fruits of your labour start to bear fruit!
> 
> > > The default chroot name I'm using is "coriolis". Should I change this, or
> > > perhaps provide an optional input argument?
> > 
> > ah yes that's a good idea
> Implemented.

ok i'm running it now

> Documentation's already there, may as well make it as accessible as possible
> ;)

i like the screenshot, that's going to be important validation for people.
 
> Is there anything else needed for this task, or is it complete?

yeah pretty much.  let me just confirm it... damn thing's locked up
on debootstrap at the moment, sigh.  ok restarted, underway.
Comment 27 Andrey Miroshnikov 2021-10-07 14:14:59 BST
(In reply to Luke Kenneth Casson Leighton from comment #26)

> i like the screenshot, that's going to be important validation for people.
That's how I felt when the environment hiccup was solved. Oh, and it looks nice!

> yeah pretty much.  let me just confirm it... damn thing's locked up
> on debootstrap at the moment, sigh.  ok restarted, underway.
Is this to do with the apt proxy issue you had?


In the meantime I'll continue looking at Robert's nmigen tutorials. So far the ParallelSignal conversations you had go over my head, partly due to architecture specifics, partly due to nmigen/OOP syntax.
Covered Const/Signals, fairly straightforward (coming from verilog background). Modules (at least initially) seem a bit more involved. Sometimes all the object instantiations just seem a little too verbose. However worth learning anyhow.
Comment 28 Luke Kenneth Casson Leighton 2021-10-07 16:21:18 BST
(In reply to andrey from comment #27)
> (In reply to Luke Kenneth Casson Leighton from comment #26)
> 
> > i like the screenshot, that's going to be important validation for people.
> That's how I felt when the environment hiccup was solved. Oh, and it looks
> nice!

yehyeh.  oh, the text string on console needs to say "type chip_r"
not type "arm_something_r"

> > yeah pretty much.  let me just confirm it... damn thing's locked up
> > on debootstrap at the moment, sigh.  ok restarted, underway.
> Is this to do with the apt proxy issue you had?

sigh, we can deduce that is likely, yes

> In the meantime I'll continue looking at Robert's nmigen tutorials. So far
> the ParallelSignal conversations you had go over my head, partly due to
> architecture specifics, partly due to nmigen/OOP syntax.
> Covered Const/Signals, fairly straightforward (coming from verilog
> background). Modules (at least initially) seem a bit more involved.
> Sometimes all the object instantiations just seem a little too verbose.
> However worth learning anyhow.

let me put this on-list (bugreports are topic-specific)
Comment 29 Andrey Miroshnikov 2021-10-07 16:54:22 BST
(In reply to Luke Kenneth Casson Leighton from comment #28)
> yehyeh.  oh, the text string on console needs to say "type chip_r"
> not type "arm_something_r"
Oops, missed that. Fixed and pushed out now. Made it into a variable "example_cell" just to make it stand out more for future changes. Perhaps there's a way to extract the name of the cell from the files in the given directory (alliance-check-toolkit/benchs/adder/cmos/)?
Although I don't know if every chip will have "_r" in the name.