Noticed the reason why "coriolisEnv.py" fails when executed from shell script.

The python script calls the unix utility "which" with an argument of the shell.
The command looks like this:
whichCommand  = subprocess.Popen ( ['which', shell ], stdout=subprocess.PIPE )

Where "shell" is determined by:
shell         = psCommand.stdout.readlines()[1][:-1].split()[3].lstrip('-')

When ran manually, "shell" returned is:
Shell: bash
which is what we want.

When ran inside a shell script, the name of the script is returned instead! At the moment I'm using a test script called "test-coriolisenv" and the value returned is the first 15 chars of the script name:
Shell: test-coriolisen


The method stdout.readline() of "whichCommand" is correct when ran manually (the path to bash):
['/usr/bin/bash\n']
And empty when inside the script (because there's no programme called "test-coriolisen" hahaha!)
[]


I'm happy at least that I identified the issue, now I need to think how to fix it. Certainly renaming "coriolis-install" to "bash" wouldn't be very prudent. I'll look further into it tomorrow.

(In reply to andrey from comment #1)
> Noticed the reason why "coriolisEnv.py" fails when executed from shell
> script.
> 
> The python script calls the unix utility "which" with an argument of the
> shell.

and we didn't encounter this before because the chroot did this:

chroot /bin/bash << EOF
coriolisEnv.py > $HOME/coriolisenv
EOF

which *is* running /bin/bash.  whoops.

https://gitlab.lip6.fr/vlsi-eda/coriolis/-/issues/48

andrey a workaround is to do "/bin/bash -c coriolisEnv.py >" blah blah

or use the EOF trick again from comment 2:

/bin/bash << EOF
.... coriolisEnv.py > blahblah
EOF

Andrey, this one we can put as a "documentation" budget,
if you can update the tutorials and pages involved that
explain how to use these two scripts.

also, Jean-Paul will be doing a coriolis_install_python3 which
should also be mentioned.

(In reply to andrey from comment #1)
> Noticed the reason why "coriolisEnv.py" fails when executed from shell
> script.
> 
> I'm happy at least that I identified the issue, now I need to think how to
> fix it. Certainly renaming "coriolis-install" to "bash" wouldn't be very
> prudent. I'll look further into it tomorrow.

  Whoops. Didn't foresee this use-case. The problem to solve here
  is to reliably find the shell the script is running under so it
  can select the right syntax to issue environment commands.
  For various Linux/UNIX and MacOS...

  Maybe the answer is "we cannot" and we should force the user to
  supply it as an argument to the coriolisEnv.py script.

(In reply to Jean-Paul.Chaput from comment #5)

>   Maybe the answer is "we cannot" and we should force the user to
>   supply it as an argument to the coriolisEnv.py script.

https://unix.stackexchange.com/questions/352316/finding-out-the-default-shell-of-a-user-within-a-shell-script

$SHELL.

(In reply to Luke Kenneth Casson Leighton from comment #6)
> (In reply to Jean-Paul.Chaput from comment #5)
> 
> >   Maybe the answer is "we cannot" and we should force the user to
> >   supply it as an argument to the coriolisEnv.py script.
> 
> https://unix.stackexchange.com/questions/352316/finding-out-the-default-
> shell-of-a-user-within-a-shell-script
> 
> $SHELL.

Of course that was my first idea... But, what if you do that,
assuming that your login shell is /bin/bash :

bash> echo $SHELL
/bin/bash
bash> tcsh
tcsh> echo $SHELL
/bin/bash

And this is a real use case. As users usually are under bash,
*but* VLSI CAD tools very much like C-Shell.

(In reply to Jean-Paul.Chaput from comment #7)

> And this is a real use case. As users usually are under bash,
> *but* VLSI CAD tools very much like C-Shell.

ok so it's the shell you are running *at the time* that is needed,
not the user's login shell.

which they don't change in /etc/passwd.

sigh.

https://www.google.com/search?q=reliably+get+current+shell+linu

Use the following Linux or Unix commands:
ps -p $$ – Display your current shell name reliably.

apparently.

(In reply to Luke Kenneth Casson Leighton from comment #9)
> https://www.google.com/search?q=reliably+get+current+shell+linu
> 
> Use the following Linux or Unix commands:
> ps -p $$ – Display your current shell name reliably.
> 
> apparently.

  Ha ha. We are going back to what coriolisEnv.py is doing (although
  it is a little bit more contrived).

  This command, if run inside a shell script, will return the
  name of the script, not the shell that execute it (just tried).

(In reply to Jean-Paul.Chaput from comment #10)

>   Ha ha. We are going back to what coriolisEnv.py is doing (although
>   it is a little bit more contrived).
> 
>   This command, if run inside a shell script, will return the
>   name of the script, not the shell that execute it (just tried).

ooooooOOO so annoying.

sounds like "coriolisEnv.py --shell `ps -p $$`" then.

or, hunt back from parent process parent process parent process until
hitting a shell.

(In reply to Luke Kenneth Casson Leighton from comment #12)
> or, hunt back from parent process parent process parent process until
> hitting a shell.

  Yes, that was also my conclusion *if* we want to keep the
  auto-detection... Sight!

(In reply to Luke Kenneth Casson Leighton from comment #3)
> https://gitlab.lip6.fr/vlsi-eda/coriolis/-/issues/48
> 
> andrey a workaround is to do "/bin/bash -c coriolisEnv.py >" blah blah
> 
> or use the EOF trick again from comment 2:
> 
> /bin/bash << EOF
> .... coriolisEnv.py > blahblah
> EOF
Good idea, I'll try both of those.

In my test script had the following (modified coriolisEnv.py to only print the shell):
# First method
/bin/bash -c /home/$USER/coriolis-2.x/src/coriolis/bootstrap/coriolisEnv.py
# Second method
/bin/bash << EOF
/home/$USER/coriolis-2.x/src/coriolis/bootstrap/coriolisEnv.py
EOF

The standard output:
Test script for checking coriolisEnv.py!
Shell: test-coriolisen
whichCommand.stdout.readlines(): []
Shell: bash
whichCommand.stdout.readlines(): ['/usr/bin/bash\n']

Section on "-c" in the bash man pages:
"If the -c option is present, then commands are read from  the
first non-option argument command_string.  If there are argu-
ments after the command_string, the  first  argument  is  as-
signed  to $0 and any remaining arguments are assigned to the
positional parameters.  The assignment to $0 sets the name of
the shell, which is used in warning and error messages."

Is "/bin/bash -c" not going to work because of $0? I tried adding "bash" after the python script (/bin/bash -c .../coriolisEnv.py bash) to see if that would set $0, but didn't work.

So the first method doesn't work, however the second does, so I'll stick to the second one for now.



(In reply to Luke Kenneth Casson Leighton from comment #4)
> Andrey, this one we can put as a "documentation" budget,
> if you can update the tutorials and pages involved that
> explain how to use these two scripts.
Sure. Is it just the coriolis2 wiki page I need to work on?
https://libre-soc.org/HDL_workflow/coriolis2/

There's also the devscripts wiki:
https://libre-soc.org/HDL_workflow/devscripts/
where I can include brief descriptions of the other scripts. Is that worth doing?

> also, Jean-Paul will be doing a coriolis_install_python3 which
> should also be mentioned.
Sure, I'll mention what needs to be updated to work with python3. Now that we found the causes of my issues, a python3 version *might* work, so I'll create a second script called "coriolis-install-py3" (with a disclaimer saying it probably doesn't work yet) after I'm done with "coriolis-install" and updated the documentation.


(In reply to Jean-Paul.Chaput from comment #13)
> (In reply to Luke Kenneth Casson Leighton from comment #12)
> > or, hunt back from parent process parent process parent process until
> > hitting a shell.
> 
>   Yes, that was also my conclusion *if* we want to keep the
>   auto-detection... Sight!
Well, at least we can use the trick Luke mentioned in comment #3 for now.

(In reply to Luke Kenneth Casson Leighton from comment #11)
> >   This command, if run inside a shell script, will return the
> >   name of the script, not the shell that execute it (just tried).
> 
> ooooooOOO so annoying.
> 
> sounds like "coriolisEnv.py --shell `ps -p $$`" then.
Also tried:
.../coriolisEnv.py --shell `ps -p $$`
didn't work either (also returned name of the script).

(In reply to andrey from comment #15)

> Also tried:
> .../coriolisEnv.py --shell `ps -p $$`
> didn't work either (also returned name of the script).

it won't - that was a prototype idea. coriolisEnv.py does not take
an option "--shell".

(In reply to andrey from comment #14)

> (In reply to Luke Kenneth Casson Leighton from comment #4)

> Sure. Is it just the coriolis2 wiki page I need to work on?
> https://libre-soc.org/HDL_workflow/coriolis2/

yes.
 
> There's also the devscripts wiki:
> https://libre-soc.org/HDL_workflow/devscripts/
> where I can include brief descriptions of the other scripts. Is that worth
> doing?

hmmm, that one i think should be kept very brief.

andrey, why is there an identical script, one with a minus symbol
and one with an underscore?

lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u coriolis-install coriolis_install 
lkcl@fizzy:~/src/libresoc/dev-env-setup$

(and also an identical one cp-scripts-to-chroot and .sh)

lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u cp-scripts-to-chroot cp-scripts-to-chroot.sh 
lkcl@fizzy:~/src/libresoc/dev-env-setup$

(In reply to Luke Kenneth Casson Leighton from comment #17)
> andrey, why is there an identical script, one with a minus symbol
> and one with an underscore?
> 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u coriolis-install
> coriolis_install 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$
> 
> (and also an identical one cp-scripts-to-chroot and .sh)
> 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$ diff -u cp-scripts-to-chroot
> cp-scripts-to-chroot.sh 
> lkcl@fizzy:~/src/libresoc/dev-env-setup$

The original script was "coriolis_install" with an underscore (my mistake), and I used mv to change the name (to better fit the rest of the scripts). Same goes for "cp-scripts-to-chroot.sh" where I removed the .sh extension.

Unfortunately git only deleted the old scripts in my local files, not on the repo. When I do a git pull, the old versions don't get downloaded. Should I use rm next time instead of mv?

(In reply to andrey from comment #18)
> (In reply to Luke Kenneth Casson Leighton from comment #17)
> > andrey, why is there an identical script, one with a minus symbol
> > and one with an underscore?
> > 
> > (and also an identical one cp-scripts-to-chroot and .sh)
> 
> Unfortunately git only deleted the old scripts in my local files, not on the
> repo. When I do a git pull, the old versions don't get downloaded.

Had to reset my local repo to the latest with
git reset --hard HEAD
and the duplicate files appeared, which I now removed. Checked web view of git and old files no longer there.

(In reply to andrey from comment #18)

> The original script was "coriolis_install" with an underscore (my mistake),
> and I used mv to change the name 

you should use "git mv" for that situation.

there is a fundamental difference between the git repository
and the "index" which is the checked-out contents that happen
to be on *your* local filesystem.

the two should not be confused or conflated.

(In reply to andrey from comment #18)

> Unfortunately git only deleted the old scripts in my local files, not on the
> repo. When I do a git pull, the old versions don't get downloaded. Should I
> use rm next time instead of mv?

the local command that is part of bash known as "rm"
would simply remove a local file.

running the local command "rm" would do absolutely nothing to the
git epository.

if however you meant *GIT RM* then obviously that is completely
different from *GIT MV*.  i leave it to you to look up the man
page on git rm and git mv?

(In reply to Luke Kenneth Casson Leighton from comment #16)
> (In reply to andrey from comment #15)
> 
> > Also tried:
> > .../coriolisEnv.py --shell `ps -p $$`
> > didn't work either (also returned name of the script).
> 
> it won't - that was a prototype idea. coriolisEnv.py does not take
> an option "--shell".

  The latest version (Python 3) now has it.

I've made a top-level script that sets up the chroot and Coriolis2 (as outlined in comment #1). Ran it once (after making sure "coriolis-install" already worked), and it works (from creating chroot all the way to displaying the chip_r floorplan)!


The default chroot name I'm using is "coriolis". Should I change this, or perhaps provide an optional input argument?


The script also checks if a chroot exists, and if it does, exits before doing anything. This is because I don't expect a user to be re-running the script more than once.
Once the chroot is setup, "coriolis-install" can be run independently.

The wiki pages for Coriolis2 and devscripts have been updated (I moved the chroot information there as it is applicable to other software). Have a go running the script "coriolis2-chroot" if you have time (probably before going off for lunch or something).


Also after checking the color codes used in mk-deb-chroot, I found a convenient way to change echo text colour:
RED='\033[1;91m'
NC='\033[0m'
echo -e "${RED}This text is in red,${NC} while this text is plain (no colour)."

There are even more options to look at:
http://www.cplusplus.com/forum/unices/36461/
https://stackoverflow.com/questions/5947742/how-to-change-the-output-color-of-echo-in-linux

"coriolis2-chroot" uses these colour variables for some of the messages.

(In reply to andrey from comment #23)
> I've made a top-level script that sets up the chroot and Coriolis2 (as
> outlined in comment #1). Ran it once (after making sure "coriolis-install"
> already worked), and it works (from creating chroot all the way to
> displaying the chip_r floorplan)!

hooraaay :)

> 
> The default chroot name I'm using is "coriolis". Should I change this, or
> perhaps provide an optional input argument?

ah yes that's a good idea
 
> 
> The script also checks if a chroot exists, and if it does, exits before
> doing anything. This is because I don't expect a user to be re-running the
> script more than once.
> Once the chroot is setup, "coriolis-install" can be run independently.

good idea
 
> The wiki pages for Coriolis2 and devscripts have been updated (I moved the
> chroot information there as it is applicable to other software).

i saw that.  nice touch about referring to documentation in the script.
i'm glad you kept the original "manual" instructions, it helps for
people who want to know what is going on.

>  Have a go
> running the script "coriolis2-chroot" if you have time (probably before
> going off for lunch or something).

well, i already have a chroot named coriolis (and use it) so that would
clearly be bad :) and would justify the extra argument.

> 
> 
> Also after checking the color codes used in mk-deb-chroot, I found a
> convenient way to change echo text colour:
> RED='\033[1;91m'
> NC='\033[0m'
> echo -e "${RED}This text is in red,${NC} while this text is plain (no
> colour)."

nice.

(In reply to Luke Kenneth Casson Leighton from comment #24)
> hooraaay :)
Indeed, it's nice to see fruits of your labour start to bear fruit!

> > The default chroot name I'm using is "coriolis". Should I change this, or
> > perhaps provide an optional input argument?
> 
> ah yes that's a good idea
Implemented.
 
> > The wiki pages for Coriolis2 and devscripts have been updated (I moved the
> > chroot information there as it is applicable to other software).
> 
> i saw that.  nice touch about referring to documentation in the script.
> i'm glad you kept the original "manual" instructions, it helps for
> people who want to know what is going on.
Documentation's already there, may as well make it as accessible as possible ;)

> >  Have a go
> > running the script "coriolis2-chroot" if you have time (probably before
> > going off for lunch or something).
> 
> well, i already have a chroot named coriolis (and use it) so that would
> clearly be bad :) and would justify the extra argument.
Now should be possible, I tested it the optional argument with just setting up the chroot.


Is there anything else needed for this task, or is it complete?

(In reply to andrey from comment #25)
> (In reply to Luke Kenneth Casson Leighton from comment #24)
> > hooraaay :)
> Indeed, it's nice to see fruits of your labour start to bear fruit!
> 
> > > The default chroot name I'm using is "coriolis". Should I change this, or
> > > perhaps provide an optional input argument?
> > 
> > ah yes that's a good idea
> Implemented.

ok i'm running it now

> Documentation's already there, may as well make it as accessible as possible
> ;)

i like the screenshot, that's going to be important validation for people.
 
> Is there anything else needed for this task, or is it complete?

yeah pretty much.  let me just confirm it... damn thing's locked up
on debootstrap at the moment, sigh.  ok restarted, underway.

(In reply to Luke Kenneth Casson Leighton from comment #26)

> i like the screenshot, that's going to be important validation for people.
That's how I felt when the environment hiccup was solved. Oh, and it looks nice!

> yeah pretty much.  let me just confirm it... damn thing's locked up
> on debootstrap at the moment, sigh.  ok restarted, underway.
Is this to do with the apt proxy issue you had?


In the meantime I'll continue looking at Robert's nmigen tutorials. So far the ParallelSignal conversations you had go over my head, partly due to architecture specifics, partly due to nmigen/OOP syntax.
Covered Const/Signals, fairly straightforward (coming from verilog background). Modules (at least initially) seem a bit more involved. Sometimes all the object instantiations just seem a little too verbose. However worth learning anyhow.

(In reply to andrey from comment #27)
> (In reply to Luke Kenneth Casson Leighton from comment #26)
> 
> > i like the screenshot, that's going to be important validation for people.
> That's how I felt when the environment hiccup was solved. Oh, and it looks
> nice!

yehyeh.  oh, the text string on console needs to say "type chip_r"
not type "arm_something_r"

> > yeah pretty much.  let me just confirm it... damn thing's locked up
> > on debootstrap at the moment, sigh.  ok restarted, underway.
> Is this to do with the apt proxy issue you had?

sigh, we can deduce that is likely, yes

> In the meantime I'll continue looking at Robert's nmigen tutorials. So far
> the ParallelSignal conversations you had go over my head, partly due to
> architecture specifics, partly due to nmigen/OOP syntax.
> Covered Const/Signals, fairly straightforward (coming from verilog
> background). Modules (at least initially) seem a bit more involved.
> Sometimes all the object instantiations just seem a little too verbose.
> However worth learning anyhow.

let me put this on-list (bugreports are topic-specific)

(In reply to Luke Kenneth Casson Leighton from comment #28)
> yehyeh.  oh, the text string on console needs to say "type chip_r"
> not type "arm_something_r"
Oops, missed that. Fixed and pushed out now. Made it into a variable "example_cell" just to make it stand out more for future changes. Perhaps there's a way to extract the name of the cell from the files in the given directory (alliance-check-toolkit/benchs/adder/cmos/)?
Although I don't know if every chip will have "_r" in the name.