Found a useful link for making git auto-check-out when nmutil.git is pushed to (in case ikiwiki needs nmutil.git to be checked out for it to work):
https://stackoverflow.com/a/10513684

>  In Git since 2.3, you can now configure receive.denyCurrentBranch to updateInstead

I instead used git subtree to sync libreriscv.git/openpower/sv/bitmanip to nmutil.git/src/nmutil/openpower_sv_bitmanip_in_wiki

Instructions here:
https://libre-soc.org/openpower/sv/bitmanip/git_subtree_syncing/

(In reply to Jacob Lifshay from comment #2)
> I instead used git subtree to sync libreriscv.git/openpower/sv/bitmanip to
> nmutil.git/src/nmutil/openpower_sv_bitmanip_in_wiki
> 
> Instructions here:
> https://libre-soc.org/openpower/sv/bitmanip/git_subtree_syncing/

please rwmove all subtrees immediately.

please remove that page.

please put a notice in HDL_workflow that subtrees are not to
be used under any circumstances.

the cross poisoning of commit messages from irrelevant
git logs is far too great.

(In reply to Luke Kenneth Casson Leighton from comment #3)
> the cross poisoning of commit messages from irrelevant
> git logs is far too great.

git subtree has a squash mode (which I didn't use) where it doesn't copy all the commits, instead just copies the changes since the last sync all as one commit. basically a slightly fancier version of cp -r and git add and git commit, rather than ending up with an entire duplicate commit chain.

(In reply to Jacob Lifshay from comment #4)
> (In reply to Luke Kenneth Casson Leighton from comment #3)
> > the cross poisoning of commit messages from irrelevant
> > git logs is far too great.
> 
> git subtree has a squash mode (which I didn't use) where it doesn't copy all

when under severe time and other pressure i am not interested in hearing
about features of something that is irrelevant.

simple question: have you complied with the instructions given to you, yes or no.

i have told you that git subtree is not to be used.

i do not have time to go into detail

you have ignored my instructions, wasting my time when i have
a critical contractual obligation to fulfil and only days in
which to complete it

please for god's sake just do as i ask.

i have once again been forced to do a force-master push
because you have not listened to what i am telling you
right when i do not have the time to go through explanations.

i have rolled back the subtree commit

i have removed your commit rights to the wiki.

please let me know when you have rolled back to the
correct master branch which does not contain subtrees
and i will then restore your commit rights.

(In reply to Luke Kenneth Casson Leighton from comment #5)
> (In reply to Jacob Lifshay from comment #4)
> > (In reply to Luke Kenneth Casson Leighton from comment #3)
> > > the cross poisoning of commit messages from irrelevant
> > > git logs is far too great.
> > 
> > git subtree has a squash mode (which I didn't use) where it doesn't copy all
> 
> when under severe time and other pressure i am not interested in hearing
> about features of something that is irrelevant.

well, when you are not under time pressure, please consider git subtree squash mode...

(In reply to Luke Kenneth Casson Leighton from comment #6)
> please let me know when you have rolled back to the
> correct master branch which does not contain subtrees
> and i will then restore your commit rights.

yes, i reset my master branch to 7ac40ec2fbba4076f4d4ce1baa0d6ed912da6232.

(In reply to Jacob Lifshay from comment #7)

> well, when you are not under time pressure, please consider git subtree
> squash mode...

it's already considered: the answer is no. i can see the damage it does
from at least two angles.

> (In reply to Luke Kenneth Casson Leighton from comment #6)

> yes, i reset my master branch to 7ac40ec2fbba4076f4d4ce1baa0d6ed912da6232.

great. commit rights restored.

Replying to:
https://lists.libre-soc.org/pipermail/libre-soc-dev/2022-March/004630.html
On Sun, Mar 27, 2022, 04:38 Luke Kenneth Casson Leighton <lkcl@lkcl.net> wrote:
> however *as a group* my feeling is that they qualify for "new repo" status.
> they're all sufficiently general that others may wish to use them.
> soc is not appropriate.  ieee754fpu is not appropriate.  nmutil isn't
> appropriate.

I like the idea of putting them in a new repo, however we still have the same issue of how they end up on the wiki -- git submodules don't work with ikiwiki.

(In reply to Luke Kenneth Casson Leighton from comment #8)
> (In reply to Jacob Lifshay from comment #7)
> 
> > well, when you are not under time pressure, please consider git subtree
> > squash mode...
> 
> it's already considered: the answer is no. i can see the damage it does
> from at least two angles.

that doesn't tell anyone else your reasons for not liking git subtree...respond when you're *not under time pressure*. (this comes across as an arbitrary decision with no discussion, it needs to be addressed at some point soon...how about the middle of april or so?)

(In reply to Jacob Lifshay from comment #9)

> I like the idea of putting them in a new repo, however we still have the
> same issue of how they end up on the wiki -- git submodules don't work with
> ikiwiki.

anything in /var/www/openpower-isa is already an underlay. there is
one directory, the openpower one:

$ ls /var/www/openpower-isa/openpower/
isa  isafunctions  isa.mdwn  isatables  README.txt

https://docs.python.org/3/distutils/setupscript.html

it is not strictly necessary to drop code into src or lib.
parts of a package may be placed in arbitrary locations.

gf*.py may therefore be placed under the openpower directory
and a package_dir entry given to set them into the openpower.xxx
namespace.

(In reply to Jacob Lifshay from comment #10)

> that doesn't tell anyone else your reasons for not liking git
> subtree...


i already did.  the answers were complete and final.

(In reply to Luke Kenneth Casson Leighton from comment #12)
> https://docs.python.org/3/distutils/setupscript.html
> 
> it is not strictly necessary to drop code into src or lib.
> parts of a package may be placed in arbitrary locations.
> 
> gf*.py may therefore be placed under the openpower directory
> and a package_dir entry given to set them into the openpower.xxx
> namespace.

I need to use them from nmutil, not just openpower-isa. iirc you don't want nmutil to depend on openpower-isa.

how about we make a new repo, op-reference.git? (since it's useful as a reference for more than just openpower, and i'll eventually want to put more than just bitmanip stuff in it, e.g. fp->int javascript-style or utf-8 verifiers)

(In reply to Jacob Lifshay from comment #14)

> I need to use them from nmutil, not just openpower-isa. iirc you don't want
> nmutil to depend on openpower-isa.

under absolutely no circumstances is that acceptable.  it has already
been explained why.

> how about we make a new repo, op-reference.git? (since it's useful as a
> reference for more than just openpower, and i'll eventually want to put more
> than just bitmanip stuff in it, e.g. fp->int javascript-style or utf-8
> verifiers)

that'd work: it's general enough. there's a number of other things
that could move to it.  will sort that out.

(In reply to Luke Kenneth Casson Leighton from comment #13)
> (In reply to Jacob Lifshay from comment #10)
> 
> > that doesn't tell anyone else your reasons for not liking git
> > subtree...
> 
> 
> i already did.  the answers were complete and final.

I think you thought you did but didn't actually...I can't find git subtree anywhere on libre-soc.org except this bug and a stale version of the sitemap (iirc google searches all subdomains too, so it should have gotten all the mailing list and irc log too)

https://www.google.com/search?q=site%3Alibre-soc.org+%22git+subtree%22

(In reply to Luke Kenneth Casson Leighton from comment #15)
> (In reply to Jacob Lifshay from comment #14)
> 
> > I need to use them from nmutil, not just openpower-isa. iirc you don't want
> > nmutil to depend on openpower-isa.
> 
> under absolutely no circumstances is that acceptable.  it has already
> been explained why.

i'm assuming you mean having nmutil depend on openpower-isa is unacceptable. if you mean nmutil depending on op-reference is also unacceptable, please clarify.

(In reply to Jacob Lifshay from comment #17)

> i'm assuming you mean having nmutil depend on openpower-isa is unacceptable.

yes absolutely.  i explained already.  if you didn't see the message
on the list please do look for it.

> if you mean nmutil depending on op-reference is also unacceptable, please
> clarify.

... hmm now that i think about it, that's probably also not going to work,
although i really like the idea of a separate repo, nmutil cannot itself
depend on it.  the *other way round* is perfectly fine.
see the message on the mailing list as to why.

(In reply to Luke Kenneth Casson Leighton from comment #18)
> (In reply to Jacob Lifshay from comment #17)
> > if you mean nmutil depending on op-reference is also unacceptable, please
> > clarify.
> 
> ... hmm now that i think about it, that's probably also not going to work,
> although i really like the idea of a separate repo, nmutil cannot itself
> depend on it.  the *other way round* is perfectly fine.
> see the message on the mailing list as to why.

what about adding op-reference to nmutil's test dependencies, since I need it to run the tests but everyone who isn't running tests doesn't need it. If that is still unacceptable, the only options i can think of that are left are generally very poor options imho: 
either:
copy the reference functions into nmutil,
rewrite the reference functions into nmutil's tests,
put nmutil's tests somewhere other than nmutil.git, or
just don't test nmutil's functions.

(In reply to Jacob Lifshay from comment #19)

> what about adding op-reference to nmutil's test dependencies,

that sounds much better, but it's still... slightly dicey.
nmutil is intended to be completely stand-alone independent.
as in, its sole exclusive dependency absolutely has to be
nmigen and nmigen only.

> since I need
> it to run the tests but everyone who isn't running tests doesn't need it. If
> that is still unacceptable, the only options i can think of that are left
> are generally very poor options imho: 
> either:
> copy the reference functions into nmutil,
> rewrite the reference functions into nmutil's tests,
> put nmutil's tests somewhere other than nmutil.git, or
> just don't test nmutil's functions.

i am beginning to remember what i wrote on the list.

nmutil is designed for general-purpose "anything anyone might need"
pipelines, bit-reversing patterns, sign-extending, popcount, priority
picker, etc. etc.  kitchen-sink stuff

gf* operations *do not* in any way qualify as "general-purpose"

only a tiny handful of HDL engineers are going to create designs
using gf* operations.

by contrast a *Very large* percentage of HDL engineers will want
to do pipelines and bit-reversing

conclusion: nmutil is completely the wrong location to consider
adding the gf* operations.

basically the gf* ops go in their own separate repo

please can you track down and read the original message, i have
just been forced to spend time repeating what i wrote, which is
very irritating when i am under time pressure.

(In reply to Luke Kenneth Casson Leighton from comment #20)
> nmutil is designed for general-purpose "anything anyone might need"
> pipelines, bit-reversing patterns, sign-extending, popcount, priority
> picker, etc. etc.  kitchen-sink stuff
> 
> gf* operations *do not* in any way qualify as "general-purpose"

In that case, can you please create a new repo where we can put the hdl and reference python for testing? And then add it as a underlay in the wiki? This is blocking progress on the cl* and gf* ops.

may I suggest naming it openpower-hdl.git ... if you don't like that, please just pick a name and go ahead and create it, we can rename it later if we need to without blocking progress deciding on a name.

(In reply to Jacob Lifshay from comment #21)

> In that case, can you please create a new repo where we can put the hdl and
> reference python for testing? And then add it as a underlay in the wiki?
> This is blocking progress on the cl* and gf* ops.

true, but it's not damaging nmutil or the openpower-isa repos in
a disastrous fashion.

did you find the original discussion and read it?
 
> may I suggest naming it openpower-hdl.git ... if you don't like that, please
> just pick a name and go ahead and create it, we can rename it later if we
> need to without blocking progress deciding on a name.

openpower-hdl is not appropriate for general gf* hdl, i'll name
it nmigen-gf or nm-gf.

(In reply to Luke Kenneth Casson Leighton from comment #22)
> (In reply to Jacob Lifshay from comment #21)
> 
> > In that case, can you please create a new repo where we can put the hdl and
> > reference python for testing? And then add it as a underlay in the wiki?
> > This is blocking progress on the cl* and gf* ops.
> 
> true, but it's not damaging nmutil or the openpower-isa repos in
> a disastrous fashion.
> 
> did you find the original discussion and read it?

no, I couldn't find it.

> > may I suggest naming it openpower-hdl.git ... if you don't like that, please
> > just pick a name and go ahead and create it, we can rename it later if we
> > need to without blocking progress deciding on a name.
> 
> openpower-hdl is not appropriate for general gf* hdl, i'll name
> it nmigen-gf or nm-gf.

I was thinking we'd put more than just gf* stuff in there, it would include stuff such as the utf-8 hdl, and other random general hdl ops that don't go in soc or nmutil or openpower-isa or ieee754fpu

done, however you'll need to let me know the directory layout before
i can add an underlay.  would it be excessive to drop the entirety
of the repo in as an underlay? i don't know, maybe it's ok.
have a think and let me know a proposed directory structure that
makes sense.  in openpower-isa i split toplevel openpower dir
(underlayed) from toplevel src (not)

(In reply to Luke Kenneth Casson Leighton from comment #24)
> done, however you'll need to let me know the directory layout before
> i can add an underlay.  would it be excessive to drop the entirety
> of the repo in as an underlay? i don't know, maybe it's ok.
> have a think and let me know a proposed directory structure that
> makes sense.  in openpower-isa i split toplevel openpower dir
> (underlayed) from toplevel src (not)

ok, add src/op_reference/reference as the underlay (or just add the whole repo):
https://git.libre-soc.org/?p=op-reference.git;a=tree;f=src/op_reference/reference;h=d283a9b392288b6b45d926fd16a0359d575e2435;hb=2d16acca9a26cd4454ddcbd39a3bdaf12ddd8741

(In reply to Jacob Lifshay from comment #23)
> > 
> > did you find the original discussion and read it?
> 
> no, I couldn't find it.

mailing list.  use date listing. go by date of this bugreport.
or gmail keyword search.

> I was thinking we'd put more than just gf* stuff in there, it would include
> stuff such as the utf-8 hdl, and other random general hdl ops that don't go
> in soc or nmutil or openpower-isa or ieee754fpu

no. already made it clear: no monorepos.  one repo, one purpose.
ieee754fp is already overloaded, simd has to be moved to nmigen-simd.
utf8 goes into nmigen-utf8.

just because other projects act in a stupid fashion by smashing code
together in multi megabyte mono repos does not mean we also have to
be as stupid.

we are *not* building a single-purpose codebase.  we therefore have
to act as such by thinking through the actions and perspectives of
potential developers, users, and contributors.

this requires placing yourself in the position of each individual
and thinking through how they will react on encountering the various
codebases.

a monorepo or anything multi-purpose will be viewed as extremely
intimidating (hundreds of directories, thousands of files)
or as "something to rip apart", taking what they can get.

anyone wishing to find something very specific should be able
to find it easily with simple keyword searches and embed it
within their code *without* being forced to smash thousands of
files into their design or, much worse, propagate copies of
source files across the internet, extracted piece-meal.

by putting special-purpose HDL into self-contained repos they
are much more likely to use the entire repo as-is WITHOUT ripping
it apart (copying one or two files only) and thus more likely
to contribute without being overwhelmed.

(In reply to Luke Kenneth Casson Leighton from comment #26)
> (In reply to Jacob Lifshay from comment #23)
> > > 
> > > did you find the original discussion and read it?
> > 
> > no, I couldn't find it.
> 
> mailing list.  use date listing. go by date of this bugreport.
> or gmail keyword search.

ah, found it, i think:
https://lists.libre-soc.org/pipermail/libre-soc-dev/2022-March/004630.html

(In reply to Jacob Lifshay from comment #25)

> ok, add src/op_reference/reference as the underlay (or just add the whole
> repo):
> https://git.libre-soc.org/?p=op-reference.git;a=tree;f=src/op_reference/
> reference;h=d283a9b392288b6b45d926fd16a0359d575e2435;
> hb=2d16acca9a26cd4454ddcbd39a3bdaf12ddd8741

wait, what? argh i forgot that had been added.  i meant nmigen-gf.
i have now deleted op-reference.

again: the exact same logic applies to nmigen-gf as to nmutil. it should contain
self-contained source code (and documentation in the form of
markdown) where "oh look the unit tests and ref code can be embedded into the
libresoc wiki as an underlay"

if using op-reference then op-reference would become a critical dependence of nmigen-gf which is just as inappropriate as nmutil becoming dependent on
op-reference or openpower-isa.

nmigen-gf *can* depend on nmutil but not the other way round.

nmigen-gf must otherwise be standalone

the wiki can be *constructed* (underlay) to use nmigen-gf but must
not depend on it

soc is where everything hangs together and depends on absolutely everything.

yes, large project management is complex.

(In reply to Luke Kenneth Casson Leighton from comment #28)
> (In reply to Jacob Lifshay from comment #25)
> 
> > ok, add src/op_reference/reference as the underlay (or just add the whole
> > repo):
> 
> wait, what? argh i forgot that had been added.  i meant nmigen-gf.

ok, here then:
https://git.libre-soc.org/?p=nmigen-gf.git;a=tree;f=src/nmigen_gf/reference;h=d283a9b392288b6b45d926fd16a0359d575e2435;hb=HEAD

(In reply to Jacob Lifshay from comment #29)
> https://git.libre-soc.org/?p=nmigen-gf.git;a=tree;f=src/nmigen_gf/reference;
> h=d283a9b392288b6b45d926fd16a0359d575e2435;hb=HEAD

maybe have it so libre-soc.org/nmigen_gf/reference -> nmigen-gf.git/src/nmigen_gf/reference

(In reply to Jacob Lifshay from comment #30)
> (In reply to Jacob Lifshay from comment #29)
> > https://git.libre-soc.org/?p=nmigen-gf.git;a=tree;f=src/nmigen_gf/reference;
> > h=d283a9b392288b6b45d926fd16a0359d575e2435;hb=HEAD
> 
> maybe have it so libre-soc.org/nmigen_gf/reference ->
> nmigen-gf.git/src/nmigen_gf/reference

figured you'd want some files to test with, added:
https://git.libre-soc.org/?p=nmigen-gf.git;a=blob;f=src/nmigen_gf/reference/README.mdwn;h=98d446e7c38a8072ca75f78e44db7df310c57b19;hb=HEAD

(In reply to Jacob Lifshay from comment #31)
> (In reply to Jacob Lifshay from comment #30)
> > (In reply to Jacob Lifshay from comment #29)
> > > https://git.libre-soc.org/?p=nmigen-gf.git;a=tree;f=src/nmigen_gf/reference;
> > > h=d283a9b392288b6b45d926fd16a0359d575e2435;hb=HEAD
> > 
> > maybe have it so libre-soc.org/nmigen_gf/reference ->
> > nmigen-gf.git/src/nmigen_gf/reference
> 
> figured you'd want some files to test with, added:
> https://git.libre-soc.org/?p=nmigen-gf.git;a=blob;f=src/nmigen_gf/reference/
> README.mdwn;h=98d446e7c38a8072ca75f78e44db7df310c57b19;hb=HEAD

added a mirror for CI:
https://salsa.debian.org/Kazan-team/mirrors/nmigen-gf

(In reply to Jacob Lifshay from comment #31)

> figured you'd want some files to test with, added:
> https://git.libre-soc.org/?p=nmigen-gf.git;a=blob;f=src/nmigen_gf/reference/
> README.mdwn;h=98d446e7c38a8072ca75f78e44db7df310c57b19;hb=HEAD

exxcellent, v. helpful. now to remember how the hell i did underlays!

the underlay is all set up and working...except it only works if nmigen-gf.git was the last repo pushed to...if i push to libreriscv.git then the underlay disappears, it reappears if i then push to nmigen-gf.git.

I moved all the python files in libreriscv.git/openpower/sv/bitmanip to nmigen-gf/gf_reference

I moved the cl* and gf* files into nmigen-gf.git and added all the other misc files.

Tests pass on CI:
https://salsa.debian.org/Kazan-team/mirrors/nmigen-gf/-/jobs/2637299

only problem left is somehow the apt install is installing an entire graphical userspace (systemd, and a bunch of other junk), it takes like 7min straight just to unpack everything, idk which package is responsible...any ideas? (respond on mailing list rather than this bug)

https://git.libre-soc.org/?p=nmigen-gf.git;a=blob;f=.gitlab-ci.yml;h=5578d80616d832ccd8110b492f8f7c5a312331aa;hb=HEAD#l11

(In reply to Jacob Lifshay from comment #35)

> only problem left is somehow the apt install is installing an entire
> graphical userspace (systemd, and a bunch of other junk), it takes like 7min
> straight just to unpack everything, idk which package is responsible...any
> ideas? (respond on mailing list rather than this bug)

graphviz, xdot.

consider "python3 setup.py develop --no-deps" for nmigen
and install the dependencies manually.

(In reply to Luke Kenneth Casson Leighton from comment #36)
> (In reply to Jacob Lifshay from comment #35)
> 
> > only problem left is somehow the apt install is installing an entire
> > graphical userspace (systemd, and a bunch of other junk), it takes like 7min
> > straight just to unpack everything, idk which package is responsible...any
> > ideas? (respond on mailing list rather than this bug)
> 
> graphviz, xdot.

well...from running:
apt-cache depends --quiet --no-suggests --no-conflicts --no-breaks --no-replaces --no-enhances --recurse autoconf bison build-essential ccache clang cmake curl flex gawk git gperf graphviz libboost-program-options-dev libffi-dev libftdi-dev libgmp-dev libreadline-dev mercurial pkg-config python python3 python3-dev python3-pip python3-setuptools python3-wheel tcl-dev xdot

it turns out that systemd is installed by xdot (trimmed dependencies):
xdot
  Depends: gir1.2-gtk-3.0
gir1.2-gtk-3.0
  Depends: libgtk-3-common
libgtk-3-common
 |Depends: dconf-gsettings-backend
dconf-gsettings-backend
  Depends: dconf-service
dconf-service
  Depends: <dbus-session-bus>
    dbus-user-session
dbus-user-session
  Depends: systemd

I wouldn't have expected that, I'd expect xdot to maybe install some X11 libraries and graphviz and that's it...

> consider "python3 setup.py develop --no-deps" for nmigen
> and install the dependencies manually.

nmigen isn't the problem...

graphviz and xdot are in the SymbiYosys dependency list:
https://symbiyosys.readthedocs.io/en/latest/install.html

(In reply to Jacob Lifshay from comment #37)
 
> graphviz and xdot are in the SymbiYosys dependency list:

of course they are.

you're not going to be doing "show top" in a non-desktop
environment.

cut them out.