Bug 190 - Setup Gitlab CI Runner for Kazan on a computer
Summary: Setup Gitlab CI Runner for Kazan on a computer
Status: RESOLVED FIXED
Alias: None
Product: Libre-SOC Website
Classification: Unclassified
Component: website (show other bugs)
Version: unspecified
Hardware: All All
: Highest blocker
Assignee: Jacob Lifshay
URL:
Depends on:
Blocks:
 
Reported: 2020-02-26 17:15 GMT by Cole Poirier
Modified: 2022-07-05 04:40 BST (History)
4 users (show)

See Also:
NLnet milestone: NLnet.2021.02A.052.CryptoRouter
total budget (EUR) for completion of task and all subtasks: 1400
budget (EUR) for this task, excluding subtasks' budget: 1400
parent task for budget allocation: 775
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:
jacob=1400


Attachments
demo email (2.45 KB, text/plain)
2022-03-31 12:23 BST, Jacob Lifshay
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Cole Poirier 2020-02-26 17:15:41 GMT
Task: Figure out how to set up GitLab CI runner on a computer:

> Jacob: "I have an old 8-core AMD FX computer with 20GB of ram and a small SSD
that I'm not currently using, I've been wanting to set it up as our
own CI runner for Kazan-team on Debian Salsa, since Debian apparently
doesn't have that many spare CI runners and since CI builds will
require building LLVM (once I re-enable it) which is a really long
process."

I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory, 
500 GB ssd, rx 570 4GB desktop that I use to do some 
work on the evenings and weekends. I’d be happy to set 
this up on there if you want to use your old rig for other 
Debian CI.

> Jacob: "I can install Debian on it and give you a user account if I can get
your public SSH key."

Sounds good. Should I email the key to you, or put it on here? Additionally, is all that is needed to run "ssh-keygen -o" and complete the interactive steps? (Following the process here: https://www.keycdn.com/support/create-ssh-key).

> Jacob: "You would have to connect to the server using SSH over Tor, since I
don't have a publicly accessible IP address."

>> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?"

Jacob and Luke, which option is preferable? SSH over tor or openvpn?

> Jacob: "One requirement is to have GitLab set up such that we have to give
people explicit permission before they can run anything on it, since
I'm running it on my home network and I don't want to deal with abuse."

So to be clear, we will be using ssh over (see above) as well as an explicit request-permission grant system, for the actual running of tests/submission of jobs? Perhaps this permissions functionality for running CI jobs already exists in gitlab? I will find out once I read the installation docs.

> Jacob: "We can check with Luke what he thinks about how much money should be
assigned to this task, since this is more of a nice-to-have rather
than a requirement. I'm thinking maybe EUR 100 at most, and if it
looks like it's going to take longer than that, we should just give
up, since it's not that important."

I am interested in doing this nice to have. I don’t think a budget should be assigned to it, I’m happy to do this to help out and learn.

Jacob: "Since this is more of a sys-admin job rather than a Rust programming
job, it may not be quite what you were hoping for. There will
definitely be Rust programming jobs later."

Perfectly okay, I'm here to help in any way I can. Looking forward to doing some rust when the time comes :-)

Jacob: "Some additional requirements: Docker should be used as the backend runner."

Does it have to be docker, or can we use podman? Am I correct in my understanding that podman is the linux container utility?

Jacob: "It should be set up to block outgoing connections from the runners to
any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If possible, can you build a bash script that will install and set everything up from a fresh install of Debian, that way, it will be easier to recover from a corrupted system, install it on more computers, and publicly document the setup. Installation instructions here: https://docs.gitlab.com/12.8/runner/install/".

Yes, setting this up as a bash script sounds like an excellent idea, will be great for when something goes bad. With regard to all of the security and network configuration, it will most likely take me many tries to get right, so I'll be posting here for feedback on my configuration from time to time. And I assume both you and Luke will do a review to make sure I've set everything up so we don't get abused.
Comment 1 Jacob Lifshay 2020-02-28 23:11:17 GMT
(In reply to Cole Poirier from email)
> I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory,
> 500 GB ssd, rx 570 4GB desktop that I use to do some
> work on the evenings and weekends. I’d be happy to set
> this up on there if you want to use your old rig for other
> Debian CI.

My old computer is just sitting unused, so I think we should use it since it won't disturb you while your working.
Comment 2 Luke Kenneth Casson Leighton 2020-02-28 23:20:48 GMT
cole, chuck debian 10 on it, we can find sonething to do with it, it's a pretty mental spec, twice as powerful as my  previous laptop.

if you can also install openvpn and sshd i can connect it to my vpn
Comment 3 Jacob Lifshay 2020-02-28 23:39:36 GMT
(In reply to Cole Poirier from comment #0)
> > Jacob: "I can install Debian on it and give you a user account if I can get
> your public SSH key."
> 
> Sounds good. Should I email the key to you, or put it on here?

email it to me. Make sure you send me the id_rsa.pub and NOT the id_rsa, since id_rsa is the private key which allows anyone who has it to impersonate you.

> Additionally,
> is all that is needed to run "ssh-keygen -o" and complete the interactive
> steps? (Following the process here:
> https://www.keycdn.com/support/create-ssh-key).

All you need is to run "ssh-keygen -o" and follow the prompts.
You should use a non-blank passphrase when you setup your SSH key, since that makes it harder for someone who somehow obtained your private key to impersonate you, since they then have to guess your passphrase before they can use it.

> 
> > Jacob: "You would have to connect to the server using SSH over Tor, since I
> don't have a publicly accessible IP address."
> 
> >> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?"
> 
> Jacob and Luke, which option is preferable? SSH over tor or openvpn?

OpenVPN is preferred, I just have to research how to set that up first.

> > Jacob: "One requirement is to have GitLab set up such that we have to give
> people explicit permission before they can run anything on it, since
> I'm running it on my home network and I don't want to deal with abuse."
> 
> So to be clear, we will be using ssh over (see above) as well as an explicit
> request-permission grant system, for the actual running of tests/submission
> of jobs? Perhaps this permissions functionality for running CI jobs already
> exists in gitlab? I will find out once I read the installation docs.

SSH is used for you to run commands on the computer in order to do initial setup. GitLab has a different system that will run automatically once it's set up.

The permission system is totally separate -- on Debian Salsa itself -- probably the Developer permission.

> > Jacob: "We can check with Luke what he thinks about how much money should be
> assigned to this task, since this is more of a nice-to-have rather
> than a requirement. I'm thinking maybe EUR 100 at most, and if it
> looks like it's going to take longer than that, we should just give
> up, since it's not that important."
> 
> I am interested in doing this nice to have. I don’t think a budget should be
> assigned to it, I’m happy to do this to help out and learn.

:)

> Jacob: "Since this is more of a sys-admin job rather than a Rust programming
> job, it may not be quite what you were hoping for. There will
> definitely be Rust programming jobs later."
> 
> Perfectly okay, I'm here to help in any way I can. Looking forward to doing
> some rust when the time comes :-)
> 
> Jacob: "Some additional requirements: Docker should be used as the backend
> runner."
> 
> Does it have to be docker, or can we use podman? Am I correct in my
> understanding that podman is the linux container utility?

I'm not familiar with podman, I just know that Docker is supported as a backend and is what I'm currently using to run CI jobs on GitHub Actions. Also, Docker has some degree of isolation to make it harder to intentionally/accidentally mess up the server whereas some of the other GitLab Runner backends may not.

> Jacob: "It should be set up to block outgoing connections from the runners to
> any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If
> possible, can you build a bash script that will install and set everything
> up from a fresh install of Debian, that way, it will be easier to recover
> from a corrupted system, install it on more computers, and publicly document
> the setup. Installation instructions here:
> https://docs.gitlab.com/12.8/runner/install/".
> 
> Yes, setting this up as a bash script sounds like an excellent idea, will be
> great for when something goes bad. With regard to all of the security and
> network configuration, it will most likely take me many tries to get right,
> so I'll be posting here for feedback on my configuration from time to time.
> And I assume both you and Luke will do a review to make sure I've set
> everything up so we don't get abused.

I will try my best :)
Comment 4 Luke Kenneth Casson Leighton 2020-02-29 00:40:35 GMT
(In reply to Jacob Lifshay from comment 

> > Jacob and Luke, which option is preferable? SSH over tor or openvpn?
> 
> OpenVPN is preferred, I just have to research how to set that up first.

it's a total pain in the neck.

so 13 years ago i automated it via a wget cgi-bin script that creates an entire new config on-demand on the server and allows a .tgz archive to be downloaded which can be unpacked on the client.

it takes 30 seconds to join a new client and be up and running in 45 seconds.

don't waste precious time on it is what i am saying.
Comment 5 Jacob Lifshay 2020-03-05 05:55:06 GMT
Got my new cheap case (thought I shouldn't just keep using a cardboard box) and finished assembling the build server.

Currently installing debian on it, after which I will fix the fan controls because it has a small fan that currently runs full-blast and I have it stuck in my bedroom.

Specs:
AMD FX-8300
20GB RAM
Geforce GT710
Gigabyte ga990-fxa-ud7 motherboard
120GB SATA SSD
430W power supply

110MB/s down 12MB/s up network
Comment 6 Jacob Lifshay 2020-03-05 09:12:02 GMT
(In reply to Jacob Lifshay from comment #5)
> Currently installing debian on it, after which I will fix the fan controls
> because it has a small fan that currently runs full-blast and I have it
> stuck in my bedroom.

got fancontrol all set up!

Can figure out openvpn stuff tomorrow.
Comment 7 Luke Kenneth Casson Leighton 2020-03-06 00:47:05 GMT
(In reply to Jacob Lifshay from comment #6)

> got fancontrol all set up!

excellent
 
> Can figure out openvpn stuff tomorrow.

okaaay, run "sftp -P922 jacob@libre-riscv.org" using one of the two ssh keys you sent me, and you'll find there's a jacob.tgz archive.  unpack it somewhere then copy the contents to /etc/openvpn

you should then be able to see http://10.6.0.1

you *might* not be able to ping 10.6.0.1, i don't know why.  annoying because it is my normal way of checking if the connection is up aaaand of course it was failing.  bizarrely, traceroute works greeeeat.  *sigh*
Comment 8 Jacob Lifshay 2020-03-06 02:25:35 GMT
I tried Luke's openvpn setup, it didn't work. I got tor working, so, Cole (and Luke), I'm ready for you to send me your SSH public key (by private email).

To use it (from Debian):
sudo apt install tor
sudo service tor start
torsocks ssh <your-user-name>@<my-address.onion>

I'll send you the correct username and address once I setup your account (if the address isn't public, it's harder to attack).

Install process was really simple:
sudo apt install tor
sudo nano /etc/tor/torrc

add/uncomment following 2 lines in "location-hidden services" section:
HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 22 127.0.0.1:22

restart service:
sudo service tor restart

inspect hidden service host name:
sudo less /var/lib/tor/hidden_service/hostname
Comment 9 Jacob Lifshay 2020-03-06 02:26:12 GMT
The above instructions are assuming that ssh is already installed and working
Comment 10 Luke Kenneth Casson Leighton 2020-03-06 09:37:31 GMT
sorry, jacob, i didn't make it clear: i can't use tor (or gpg except for
signing rather than encryption).  strong encryption for me is out.
Comment 11 Luke Kenneth Casson Leighton 2020-03-06 10:19:57 GMT
btw you and cole will be fine using tor.
Comment 12 Jacob Lifshay 2020-03-10 22:47:34 GMT
Forgot to mention earlier that we got it working where you can ssh to the build server through libre-riscv.org
Comment 13 Luke Kenneth Casson Leighton 2020-03-11 01:52:48 GMT
yes. sorry! v busy.  ssh key is all that's needed, we worked out how to automatically ssh tunnel over a VPN link so it looks like the public connection is to jacob's private machine when it's not :)

so build server is up. who is configuring it? whoever it is, send me ssh public key and preferred username.
Comment 14 Jacob Lifshay 2020-07-21 22:51:13 BST
status update: as far as I know, the gitlab build daemon is installed and connected to debian salsa.

the scripts I wrote to automatically mirror git commits from git.libre-soc.org work, however they are currently disabled because they push to salsa once a minute even if there aren't any new commits and that was causing problems for debian salsa.

We *really* should just add a post-receive hook to the repos in git.libre-soc.org to automatically push to debian salsa instead of using the scripts I wrote which depend on polling once per minute. We can add libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing libre-soc to push without requiring a user account on debian salsa.

The Rust program to copy the build logs from debian salsa to a git repo works but is not deployed. the shell scripts that send out notification emails need to be written, they are ran by that rust program.

the build log mailing list needs to be setup.
Comment 15 Cole Poirier 2020-07-21 23:04:17 BST
(In reply to Jacob Lifshay from comment #14)
> status update: as far as I know, the gitlab build daemon is installed and
> connected to debian salsa.
> 
> the scripts I wrote to automatically mirror git commits from
> git.libre-soc.org work, however they are currently disabled because they
> push to salsa once a minute even if there aren't any new commits and that
> was causing problems for debian salsa.
> 
> We *really* should just add a post-receive hook to the repos in
> git.libre-soc.org to automatically push to debian salsa instead of using the
> scripts I wrote which depend on polling once per minute. We can add
> libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing
> libre-soc to push without requiring a user account on debian salsa.
> 
> The Rust program to copy the build logs from debian salsa to a git repo
> works but is not deployed. the shell scripts that send out notification
> emails need to be written, they are ran by that rust program.
> 
> the build log mailing list needs to be setup.

Thanks Jacob, that's very helpful. Can you break this down into a list of tasks? i.e. what needs to be done first and how you recommend I should do it? I would do this myself, but I don't have your knowledge.
Comment 16 Jacob Lifshay 2020-07-21 23:22:35 BST
(In reply to Cole Poirier from comment #15)
> Thanks Jacob, that's very helpful. Can you break this down into a list of
> tasks? i.e. what needs to be done first and how you recommend I should do
> it? I would do this myself, but I don't have your knowledge.

1. Add post-receive git hooks to git.libre-soc.org that mirror repos to debian salsa.

2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
create config.toml file based on
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-config.toml

and change the email.command entry to something like:
[email]
command = ["/path/to/send-email.sh"]

and create a shell script send-email.sh containing the commands to create and send the emails as required. All of the values it needs to know are passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the build log is passed in on the standard input -- so you'd probably want to run `tail -n 50` to get the last 50 lines of the build log to include in the email. the full build log is included in the git commit to the build archiving repo, so you don't need to save all of it.
Comment 17 Jacob Lifshay 2020-07-21 23:24:02 BST
you (or more likely Luke or Alain) would setup the mailing list as part of step 2
Comment 18 Cole Poirier 2020-07-21 23:27:40 BST
(In reply to Jacob Lifshay from comment #17)
> you (or more likely Luke or Alain) would setup the mailing list as part of
> step 2

That's great, thanks for taking the time to lay it out so nicely for me Jacob. Hmm I guess it would be ci.libre-soc.org?
Comment 19 Jacob Lifshay 2020-07-21 23:35:32 BST
(In reply to Cole Poirier from comment #18)
> (In reply to Jacob Lifshay from comment #17)
> > you (or more likely Luke or Alain) would setup the mailing list as part of
> > step 2
> 
> That's great, thanks for taking the time to lay it out so nicely for me
> Jacob. Hmm I guess it would be ci.libre-soc.org?

I would expect the mailing list to be something like libre-soc-builds@lists.libre-soc.org

The CI server is jacob-build-server.programmerjake.tk, which is not publicly accessible.
Comment 20 Cole Poirier 2020-07-23 19:40:36 BST
(In reply to Jacob Lifshay from comment #16)
> 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to
> debian salsa.

Would this just be a bash file with the path .git/hooks/post-receive on the libre-soc git server with the following contents?

```
#!/bin/bash
git push salsa -f --mirror
```
Comment 21 Luke Kenneth Casson Leighton 2020-07-23 19:49:35 BST
(In reply to Cole Poirier from comment #20)
> (In reply to Jacob Lifshay from comment #16)
> > 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to
> > debian salsa.
> 
> Would this just be a bash file with the path .git/hooks/post-receive on the
> libre-soc git server with the following contents?

given that i often have a commit rate of 10 to 30 commits per day this still results in too large a load on debian's servers.

it would be better pushed to the other machine behind the VPN.
Comment 22 Jacob Lifshay 2020-07-23 19:55:17 BST
(In reply to Luke Kenneth Casson Leighton from comment #21)
> given that i often have a commit rate of 10 to 30 commits per day this still
> results in too large a load on debian's servers.

I think you may be overestimating the load -- remember we're using my server to do the actual CI builds, debian salsa is just coordinating. This also has the benefit that people can view the source on the mirror if they prefer gitlab's interface.
Comment 23 Luke Kenneth Casson Leighton 2020-07-23 19:58:18 BST
(In reply to Jacob Lifshay from comment #22)

> I think you may be overestimating the load -- remember we're using my server
> to do the actual CI builds,

ah great, i didn't realise.
Comment 24 Cole Poirier 2020-07-23 20:17:55 BST
(In reply to Jacob Lifshay from comment #22)
> (In reply to Luke Kenneth Casson Leighton from comment #21)
> > given that i often have a commit rate of 10 to 30 commits per day this still
> > results in too large a load on debian's servers.
> 
> I think you may be overestimating the load -- remember we're using my server
> to do the actual CI builds, debian salsa is just coordinating. This also has
> the benefit that people can view the source on the mirror if they prefer
> gitlab's interface.

Is my solution correct and I should move on to working on the email part, and you either Luke or Jacob will add this file to the libre-soc git repo?
Comment 25 Jacob Lifshay 2020-07-23 20:21:11 BST
(In reply to Cole Poirier from comment #24)
> Is my solution correct and I should move on to working on the email part,
> and you either Luke or Jacob will add this file to the libre-soc git repo?

yes, as far as I know. Luke or someone else would have to add it since I don't have access to the server, otherwise I would have added it myself a while ago.
Comment 26 Cole Poirier 2020-07-24 23:57:46 BST
(In reply to Jacob Lifshay from comment #16)
> 2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
> create config.toml file based on
> https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-
> config.toml
> 
> and change the email.command entry to something like:
> [email]
> command = ["/path/to/send-email.sh"]
> 
> and create a shell script send-email.sh containing the commands to create
> and send the emails as required. All of the values it needs to know are
> passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the
> build log is passed in on the standard input -- so you'd probably want to
> run `tail -n 50` to get the last 50 lines of the build log to include in the
> email. the full build log is included in the git commit to the build
> archiving repo, so you don't need to save all of it.

Given that this deals with the mailing list Luke, would this be a task that should be done by Alain?
Comment 27 Cole Poirier 2020-08-27 01:36:56 BST
(In reply to Cole Poirier from comment #26)
> (In reply to Jacob Lifshay from comment #16)
> > 2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
> > create config.toml file based on
> > https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-
> > config.toml
> > 
> > and change the email.command entry to something like:
> > [email]
> > command = ["/path/to/send-email.sh"]
> > 
> > and create a shell script send-email.sh containing the commands to create
> > and send the emails as required. All of the values it needs to know are
> > passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the
> > build log is passed in on the standard input -- so you'd probably want to
> > run `tail -n 50` to get the last 50 lines of the build log to include in the
> > email. the full build log is included in the git commit to the build
> > archiving repo, so you don't need to save all of it.
> 
> Given that this deals with the mailing list Luke, would this be a task that
> should be done by Alain?

Just spoke to Jacob and he said that we can't use his build server/CI fully yet because the things he mentioned that need to be changed on our server haven't been completed yet. Luke can you remind Alain to do this along with the ISA manual latex work, and the other stuff he is doing?
Comment 28 Luke Kenneth Casson Leighton 2021-04-30 13:17:24 BST
(In reply to Jacob Lifshay from comment #19)

> I would expect the mailing list to be something like
> libre-soc-builds@lists.libre-soc.org

ah.  a mailing list.  i was expecting a simple redirector.

a mailing list will completely overwhelm the VM allocation within months, with what is effectively autogenerated output.

we cannot do that.

can you please instead set it up locally, and i will ask Alain to provide http proxy redirection.

if this was not autogenerated (reproducible) output i would not suggest this.

if bugs are noticed the output from a given run can be captured along with the git tags and a bugreport created, manually.
Comment 29 Jacob Lifshay 2021-04-30 13:53:49 BST
(In reply to Luke Kenneth Casson Leighton from comment #28)
> (In reply to Jacob Lifshay from comment #19)
> 
> > I would expect the mailing list to be something like
> > libre-soc-builds@lists.libre-soc.org
> 
> ah.  a mailing list.  i was expecting a simple redirector.
> 
> a mailing list will completely overwhelm the VM allocation within months,
> with what is effectively autogenerated output.

I'd be very surprised if that were the case, since I expect the emails to be on the order of 1-2kB each since they only contain ~60 lines of text. To use 1GB of space would take like 1,000,000 commits!

The actual build log is stored in a git repo where git is very good about compressing out redundancies across files (when objects get packed into a pack file).
Comment 30 Luke Kenneth Casson Leighton 2021-04-30 13:59:21 BST
(In reply to Jacob Lifshay from comment #29)

> I'd be very surprised if that were the case, since I expect the emails to be
> on the order of 1-2kB each since they only contain ~60 lines of text.

ok so it doesn't send the actual logs themselves as attachments.
then that's perfectly doable.
Comment 31 Jacob Lifshay 2021-04-30 14:04:23 BST
(In reply to Luke Kenneth Casson Leighton from comment #30)
> (In reply to Jacob Lifshay from comment #29)
> 
> > I'd be very surprised if that were the case, since I expect the emails to be
> > on the order of 1-2kB each since they only contain ~60 lines of text.
> 
> ok so it doesn't send the actual logs themselves as attachments.
> then that's perfectly doable.

Yup:
(comment #16)
> so you'd probably want to run `tail -n 50` to get the last 50 lines of the
> build log to include in the email. the full build log is included in the git
> commit to the build archiving repo, so you don't need to save all of it.
Comment 32 Luke Kenneth Casson Leighton 2021-08-06 20:34:16 BST
i've been thinking about this for some time, wondering what it is that makes me reticentvto action it, and i don't feel comfortable hosting large amounts of automated output (and having to back it up, and pay for the hosting, and ask Alain to host a duplicate backup).

instead please install a mailing list locally. mailman2 is trivial to set up.

if there are important bugs found the report and repro conditions may be duplicated and an appropriate bugreport raised where *ONLY* the relevant and crucial information may be recorded for aufit purposes.

even hosting "small" success reports is not useful information

closing as invalid.
Comment 33 Luke Kenneth Casson Leighton 2021-08-06 20:35:22 BST
ah this is not the mailing list one. or, it is, but the action is no longer Alain or myself.  removing Alain from the action list, a libre-soc list is no longer required.
Comment 34 Jacob Lifshay 2021-08-08 06:11:49 BST
email is still required to notify those who pushed a commit that broke the tests.
Comment 35 Luke Kenneth Casson Leighton 2021-08-08 11:04:15 BST
(In reply to Jacob Lifshay from comment #34)
> email is still required to notify those who pushed a commit that broke the
> tests.

which can be set up locally on the machine, which is part of the VPN,
and can use the local IP address of libre-soc.org 10.6.0.1 as an SMTP
relay.  installing ssmtpd would do the trick, it's extremely simple.
Comment 36 Jacob Lifshay 2022-03-29 12:14:58 BST
started working on this again after some prompting:
https://libre-soc.org/irclog/%23libre-soc.2022-03-29.log.html#t2022-03-29T08:46:00

The old script I wrote to do git mirroring didn't work very well -- the Salsa admins told me to turn it off because it was abusing their server by pushing nothing every 1 or 5min (icr which).

In order to fix that, I modified the code from
https://git.sr.ht/~janbaudisch/git-mirror
to only push to the target repo if there were changes.
I also added debian packaging and modified it to use more standard directories for debian. it runs from a systemd timer. I also added tests.

Modified repo:
https://salsa.debian.org/Kazan-team/git-mirror

git has no good mechanism that I could find to tell you if `git fetch` did anything, so I just set it to create 2 local repos and push from one to the other (using the ext protocol to call git-receive-pack -- avoiding needing to go through ssh or some other internet thing). The target repo has a post-receive hook that pushes to the final remote repo, since post-receive is only called when things change.

(all of this would be avoided if luke would just install a post-receive hook on git.libre-soc.org, but he said no iirc)

I'll install it on the build server after getting some sleep.
Comment 37 Jacob Lifshay 2022-03-29 22:49:30 BST
i forgot to set git-mirror to run as non-root (handy for having a separate ssh key, as well as security), so I'll add that next.
Comment 38 Jacob Lifshay 2022-03-30 03:58:51 BST
I got the mirrors all set up and running, the mirror scripts needed a bunch of misc. fixes...good thing I have tests now!

I moved all the mirror repos to the mirrors subgroup:
https://salsa.debian.org/Kazan-team/mirrors
Comment 39 Jacob Lifshay 2022-03-31 08:49:22 BST
I updated gitlab-ci-archiver and wrote an email-sending script, I'll try installing it tomorrow.

https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/send-email.sh

I also set up the postfix server and tested it, couldn't get it to work, afaict libre-soc.org's mail server isn't configured to let my server send anything.

Luke, can you try to take care of that if you have time? I may be able to send some test emails in an hour or so if you like. (or you can log into my build server and send some yourself, using /usr/sbin/sendmail)
Comment 40 Luke Kenneth Casson Leighton 2022-03-31 10:13:16 BST
$ telnet 10.6.0.1 25
Trying 10.6.0.1...
Connected to 10.6.0.1.
Escape character is '^]'.
421 Too many concurrent SMTP connections; please try again later.

step 1, check that's ok

step 2, reverse-dns (now added 10.6.0.7)

step 3, configure the sender correctly.

i said "make it look like libre-soc.org".
you've made it look like "build1.libre-soc.org" which fails sender-verify.
Comment 41 Luke Kenneth Casson Leighton 2022-03-31 10:46:10 BST
urr i had to actually set up reverse-dns in bind *shudder*

try this from build1 to confirm:

$ telnet 10.6.0.1 25
Trying 10.6.0.1...
Connected to 10.6.0.1.
Escape character is '^]'.
220 libre-soc.org ESMTP Exim 4.92 Thu, 31 Mar 2022 10:41:20 +0100
HELO localhost   <---- TYPE
250 libre-soc.org Hello fizzy.vpn [10.6.0.5]
MAIL FROM:lkcl@lkcl.net <---- TYPE
250 OK
RCPT TO:luke.leighton@gmail.com <----- TYPE
250 Accepted
DATA            <------ TYPE
354 Enter message, ending with "." on a line by itself
From: lkcl@lkcl.net               <------ TYPE all below
To: luke.leighton@gmail.com
Subject: test

test
.
250 OK id=1nZrJD-0007yx-K8

and configure build1 to pretend to *ONLY* be libre-soc.org
*NOT* repeat *NOT* build1.libre-soc.org
Comment 42 Jacob Lifshay 2022-03-31 10:48:34 BST
(In reply to Luke Kenneth Casson Leighton from comment #41)
> and configure build1 to pretend to *ONLY* be libre-soc.org
> *NOT* repeat *NOT* build1.libre-soc.org

i changed postfix's config to try to be libre-soc.org...i sent you a bounce i got after that.
Comment 43 Luke Kenneth Casson Leighton 2022-03-31 11:03:40 BST
yep you sent it from jacob@libre-soc.org which is a non-existent
email address, and you (legitimately) got a notification
"sender verification failed"

i've added jacob@libre-soc.org so that should now succeed.
Comment 44 Jacob Lifshay 2022-03-31 12:23:34 BST
Created attachment 160 [details]
demo email

i set gitlab-ci-archiver to work archiving all the previous pipelines, i disabled sending emails for that. (assuming there isn't any bugs)

tomorrow i'll set up the cron job or systemd timer for running it and re-enable sending emails.

any ideas for a good allow-list of who to send emails to? i want to avoid spamming random people if we import linux's source code or something.

i'm thinking go through the list of people on our budget tracking software and use that as an initial email list.

next, i'll need to make the git repo publicly visible for both https clones and gitweb (or something similar).
Comment 45 Jacob Lifshay 2022-04-01 04:15:35 BST
(In reply to Luke Kenneth Casson Leighton from comment #41)
> urr i had to actually set up reverse-dns in bind *shudder*

well, reverse-dns broke again:

jacob@jacob-build-server:~$ { printf 'HELO jacob-build-server.programmerjake.tk\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25
220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:11:28 +0100
250 libre-soc.org Hello jacob-build-server.programmerjake.tk [10.6.0.7]
250 OK
550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed)
221 libre-soc.org closing connection
jacob@jacob-build-server:~$ { printf 'HELO libre-soc.org\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25
220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:12:03 +0100
250 libre-soc.org Hello libre-soc.org [10.6.0.7]
250 OK
550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed)
221 libre-soc.org closing connection
jacob@jacob-build-server:~$ { printf 'HELO localhost\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25
220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:13:16 +0100
250 libre-soc.org Hello localhost [10.6.0.7]
250 OK
550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed)
221 libre-soc.org closing connection
Comment 46 Jacob Lifshay 2022-04-01 06:43:27 BST
I got nginx set up on my build server and confirmed that I could proxy to it using nginx on another computer.

I have an index.html for the root, and I have /build-archive.git pointed to the git repo (as a dumb git server, since that's easier to set up and since it's a totally static website).

once that's set up, you can clone:
git clone https://build.programmerjake.tk/build-archive.git

needed nginx proxy config (modify as needed for libre-soc.org):

server {
        listen 80;
        listen [::]:80;
        
        listen 443 ssl;
        listen [::]:443 ssl;
        
        server_name build.programmerjake.tk;
        ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem;
        # Redirect non-https traffic to https
        if ($scheme != "https") {
                return 301 https://$host$request_uri;
        }
                
        location / {
                proxy_pass http://jacob-build-server.programmerjake.tk/;
        }
}

Modifications needed:
* build.programmerjake.tk will need to be replaced with build.libre-soc.org
* ssl cert paths need to be adjusted
* proxy_pass needs to have the domain name replaced with 10.6.0.7
Additional steps:
* set up build.libre-soc.org to point to libre-soc.org's server
* add a ssl certificate, or re-issue the existing one but with build.libre-soc.org added as a subject alternate name.
Comment 47 Jacob Lifshay 2022-04-01 06:57:38 BST
(In reply to Jacob Lifshay from comment #46)
> * add a ssl certificate, or re-issue the existing one but with
> build.libre-soc.org added as a subject alternate name.

If you're using http[s] for letsencrypt's validation, the nginx server block will need adjusting to handle the /.well-known/acme-challenge path on libre-soc.org's server rather than proxying it.
Comment 48 Jacob Lifshay 2022-04-01 07:05:22 BST
proxying /.well-known:

server {
        listen 80;
        listen [::]:80;
        
        listen 443 ssl;
        listen [::]:443 ssl;
        
        server_name build.programmerjake.tk;
        ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem;
        # Redirect non-https traffic to https
        if ($scheme != "https") {
                return 301 https://$host$request_uri;
        }

        location /.well-known {
                root /var/www/html;
                index index.html index.htm index.nginx-debian.html;
                try_files $uri $uri/ =404;
        }
                
        location / {
                proxy_pass http://jacob-build-server.programmerjake.tk/;
        }
}
Comment 49 Luke Kenneth Casson Leighton 2022-04-01 12:07:44 BST
we need alain for that one because he created alternative
scripts (for security reasons) for acme.
Comment 50 Jacob Lifshay 2022-04-14 05:27:20 BST
I now have an fpga to add to the CI server:
https://bugs.libre-soc.org/show_bug.cgi?id=774#c21
Comment 51 Jacob Lifshay 2022-04-16 01:43:28 BST
tried passing through the tty devices, discovered that's far from sufficient:
https://libre-soc.org/irclog/%23libre-soc.2022-04-16.log.html#t2022-04-16T00:56:51
Comment 52 Jacob Lifshay 2022-04-16 01:55:44 BST
(In reply to Jacob Lifshay from comment #51)
> tried passing through the tty devices, discovered that's far from sufficient:
> https://libre-soc.org/irclog/%23libre-soc.2022-04-16.log.html#t2022-04-16T00:
> 56:51

i'll probably try luke's suggestion of running xc3sprog outside docker via ssh
Comment 53 Jacob Lifshay 2022-04-18 08:00:30 BST
did more research into passing the tty devices into docker so scripts can see the fpga's uart:

tty devices are dynamically created and removed during the fpga programming process -- docker's method of dealing with dynamic devices:
https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/man/src/container/create-example.md#dealing-with-dynamically-created-devices---device-cgroup-rule

gitlab-runner merge request for adding the device cgroup rules option:
https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309

afaict gitlab-runner uses docker through the go library rather than through the docker executable, so making a wrapper script probably won't work
Comment 54 Jacob Lifshay 2022-04-19 08:47:45 BST
I got blinky to successfully program using gitlab-runner and docker on my desktop, I'll put it on the build server tomorrow.

Took a while to get all the required stuff working (nextpnr-xilinx and all that fun stuff).

I realized that ssh wouldn't really work since the docker container wouldn't have any ssh keys that were valid for the host, and I didn't really want it to have any...I realize that could be worked around, but it sounds like a pain.

So instead, I wrote a python program that tunnels out of docker using a unix socket, the daemon outside runs xc3sprog (as well as saves a copy of the last .bit file, came in handy for testing). the unix socket can be put inside the docker container by using the gitlab-runner config to set up a docker volume that is just the unix socket file.

apparently xc3sprog wants its stdout/stderr to be a terminal, so I had to figure that out too.

I want to move program-fpga.py to another repo so we have a centralized location for it so we don't have a bunch of copies getting out of sync, does utils.git sound good?

https://salsa.debian.org/Kazan-team/gitlab-runner-test/-/commit/0447eefda9e443842e09b11672d73082a3df83c6
Comment 55 Jacob Lifshay 2022-04-20 03:56:20 BST
(In reply to Jacob Lifshay from comment #54)
> I want to move program-fpga.py to another repo so we have a centralized
> location for it so we don't have a bunch of copies getting out of sync, does
> utils.git sound good?

I'll just move it to utils.git.
Comment 56 Jacob Lifshay 2022-04-20 07:27:14 BST
I got blinky.py to work, programming a fpga successfully when run in gitlab-ci:
https://salsa.debian.org/Kazan-team/mirrors/utils/-/jobs/2685425#L7937

Other repos should add utils.git as a submodule on branch gitlab-build-utils and set XC3SPROG="$PWD/gitlab-build-utils/program-fpga.py"

https://git.libre-soc.org/?p=utils.git;a=tree;h=d6a59e33c7c197d0c61abe7f4ae5666c95535b89;hb=d6a59e33c7c197d0c61abe7f4ae5666c95535b89

On the build server, the systemd service program-fpga.service handles running xc3sprog, the socket /run/program-fpga.sock is mounted at /arty-prog-socket inside the docker containers used by gitlab-ci.
Comment 57 Jacob Lifshay 2022-04-20 08:13:48 BST
(In reply to Jacob Lifshay from comment #56)
> I got blinky.py to work, programming a fpga successfully when run in
> gitlab-ci:
> https://salsa.debian.org/Kazan-team/mirrors/utils/-/jobs/2685425#L7937
> 
> Other repos should add utils.git as a submodule on branch gitlab-build-utils
> and set XC3SPROG="$PWD/gitlab-build-utils/program-fpga.py"
> 
I added a readme with instructions (as well as instructions for installing it on the build server, in case we need them later):

Rendered:
https://salsa.debian.org/Kazan-team/mirrors/utils/-/blob/ec98be986b007dcac889d7b5077b962fe73b030a/README.md

https://git.libre-soc.org/?p=utils.git;a=blob;f=README.md;h=c219e164330c02f5fcdfedf9b8abe4e578215d19;hb=ec98be986b007dcac889d7b5077b962fe73b030a
Comment 58 Jacob Lifshay 2022-04-21 06:27:45 BST
I added a uart_demo.py that programs the FPGA with a simple transmit-only UART that's connected to all UART TX pins defined by the platform and transmits some text in a repeating loop. The default text is "Hello World!\n" and the default baud rate is 9600.

I built it with gitlab-ci and it successfully programmed the fpga, but since I don't yet have the serial port passed into the gitlab-ci docker container, I just logged onto the build server to test it...it works!

https://git.libre-soc.org/?p=utils.git;a=blob;f=uart_demo.py;h=89c40e71fc544d16e4c02464b00f20fb3d95ad05;hb=289ae5cc49eee8380210e09db3a8ec3f12ca176a

The reason I wrote the simple UART is so we have something to test that the FPGA on gitlab-ci works without needing to build a whole soft-cpu that could easily be broken by someone while their working on it...uart_demo.py is unlikely to break.

I'll add code that reads from the serial port from inside the docker container and checks that it got the right text and didn't time-out...at that point, I can declare the FPGA to be officially working on gitlab-ci...I'll wire up the hyperram later.
Comment 59 Luke Kenneth Casson Leighton 2022-04-21 07:39:05 BST
(In reply to Jacob Lifshay from comment #58)
> I added a uart_demo.py that programs the FPGA with a simple transmit-only
> UART that's connected to all UART TX pins defined by the platform and
> transmits some text in a repeating loop. The default text is "Hello
> World!\n" and the default baud rate is 9600.

ooo, that's worthwhile adding to nmigen in the same directory as
blinky (in the same style: generic code separated from platform)

> The reason I wrote the simple UART is so we have something to test that the
> FPGA on gitlab-ci works without needing to build a whole soft-cpu that could
> easily be broken by someone while their working on it...

no great idea

> uart_demo.py is unlikely to break.

yeah totally get it.

> I'll add code that reads from the serial port from inside the docker
> container and checks that it got the right text and didn't time-out...at
> that point, I can declare the FPGA to be officially working on
> gitlab-ci...I'll wire up the hyperram later.

components face-up.
Comment 60 Jacob Lifshay 2022-05-23 20:15:11 BST
(In reply to Luke Kenneth Casson Leighton from comment #49)
> we need alain for that one because he created alternative
> scripts (for security reasons) for acme.

It's been a month and a half, since it doesn't seem like alain will get around to this anytime soon, lkcl can you please work on this?
Comment 61 Luke Kenneth Casson Leighton 2022-05-23 22:24:55 BST
no, i can't. use a relative URL instead.
Comment 62 Jacob Lifshay 2022-05-23 23:15:23 BST
(In reply to Luke Kenneth Casson Leighton from comment #61)
> no, i can't. use a relative URL instead.

hmm, another option is to have the tls on my end...
https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on-hostname/40135151#40135151
Comment 63 Jacob Lifshay 2022-05-24 00:43:36 BST
(In reply to Jacob Lifshay from comment #62)
> (In reply to Luke Kenneth Casson Leighton from comment #61)
> > no, i can't. use a relative URL instead.
> 
> hmm, another option is to have the tls on my end...
> https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on-
> hostname/40135151#40135151

lkcl on irc:
https://libre-soc.org/irclog/%23libre-soc.2022-05-23.log.html#t2022-05-23T23:42:41
> i believe the SSL cert is specifically set up to prohibit that. if it was a
> completely different domain, no problem. as a subdomain, no.
> please find a relative url config (proxy pass) i can just cut/paste in

afaict the certificate is not set up to prohibit that, the tls cert has no fields which could prohibit that, you're serving no http headers which could prohibit that, and you have no dns records which could prohibit that
Comment 64 Jacob Lifshay 2022-05-27 05:28:01 BST
I have another idea for setting up the proxy for the build server:
have the proxy be on https://libre-soc.org:8443/

this successfully achieves having a different origin (which is why I wanted build.libre-soc.org).

that way it can just use the existing tls certificate for libre-soc.org.

since letsencrypt uses port 80 rather than 8443, that won't be an issue.

nginx config I used for testing (I used port 10443 because my server already was using 8443 for something else):
server {
        listen 10443 ssl;
        listen [::]:10443 ssl;
        
        ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem;
                
        location / {
                proxy_pass http://jacob-build-server.programmerjake.tk/;
                proxy_set_header Host $host;
        }
}

modifying it for libre-soc.org:
/etc/nginx/sites-available/build
server {
        listen 8443 ssl;
        listen [::]:8443 ssl;
        
        ssl_certificate /path/to/libre-soc.org.crt;
        ssl_certificate_key /path/to/libre-soc.org.key;
                
        location / {
                proxy_pass http://10.6.0.7/;
                proxy_set_header Host $host;
        }
}
Comment 65 Jacob Lifshay 2022-06-01 21:51:29 BST
(In reply to Jacob Lifshay from comment #53)
> did more research into passing the tty devices into docker so scripts can
> see the fpga's uart:
> 
> tty devices are dynamically created and removed during the fpga programming
> process -- docker's method of dealing with dynamic devices:
> https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/
> man/src/container/create-example.md#dealing-with-dynamically-created-devices-
> --device-cgroup-rule
> 
> gitlab-runner merge request for adding the device cgroup rules option:
> https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309
> 
> afaict gitlab-runner uses docker through the go library rather than through
> the docker executable, so making a wrapper script probably won't work

https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309

was merged, it should be in the next stable release at the end of this month
Comment 66 Jacob Lifshay 2022-06-02 02:46:55 BST
(In reply to Jacob Lifshay from comment #45)
> (In reply to Luke Kenneth Casson Leighton from comment #41)
> > urr i had to actually set up reverse-dns in bind *shudder*
> 
> well, reverse-dns broke again:

reverse-dns is still broken. maybe i should just see if i can get postfix to send mail through a ssh tunnel so libre-soc.org sees the connection coming from localhost
Comment 67 Jacob Lifshay 2022-06-08 08:47:45 BST
Since the openvpn connection between my build server and libre-soc.org is borked, I thought I'd come up with a better alternative:

I created a script for setting up wireguard:
https://git.libre-soc.org/?p=utils.git;a=blob;f=wg-setup.sh;h=299f5c08fc2a15112b799d69f67da132129ee322;hb=431799b73339f28ec914cea711c1af9d87533bc2

I tested it connecting my desktop and file server, trying both ways of which one is server and which one is client.

The script supports multiple clients, but I didn't test that other than seeing if it does the correct modifications to the config file.

I set it to use pre-shared keys as an extra layer of security on top of the public/private keys.

I also set it to send keep-alive packets every 25s so it works better with NATs.

Show Usage:
./wg-setup.sh

It works:
on both server and client:
sudo apt install wireguard

on the server:
# create the config for wg0 that listens on
# udp port 8995 (picked randomly) and has the server be 10.255.255.1
sudo ./wg-setup.sh init-server wg0 8995 10.255.255.1/32

# allow udp port 8995 through the server's firewall,
# here I'm using ufw, adjust as needed.
sudo ufw allow 8995/udp

# create the config for the client in client.conf, adds the peering
# info to the server's config for wg0. the client will initially
# connect to the server at server.example.com. the client
# will be 10.255.255.2.
sudo ./wg-setup.sh add-client wg0 server.example.com client.conf 10.255.255.2/32

now, move client.conf to the client, chmod 600, chown root:root, and move it to /etc/wireguard/wg0.conf

now, if you like systemd, run on both server and client:
systemctl start wg-quick@wg0

otherwise, you can start it manually:
wg-quick up wg0

or look up how wg-quick works with sysv init.

If you're using openrc (idk if that's part of sysv init), afaict you can use this init script:
https://gitweb.gentoo.org/repo/gentoo.git/tree/net-vpn/wireguard-tools/files/wg-quick.init
and symlink to it:
ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.vpn0
Comment 68 Jacob Lifshay 2022-06-08 10:37:50 BST
(In reply to Jacob Lifshay from comment #67)
> If you're using openrc (idk if that's part of sysv init), afaict you can use
> this init script:
> https://gitweb.gentoo.org/repo/gentoo.git/tree/net-vpn/wireguard-tools/files/
> wg-quick.init
> and symlink to it:
> ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.vpn0

actually, that should be .wg0:
ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.wg0
Comment 69 Jacob Lifshay 2022-06-09 03:14:14 BST
(In reply to Jacob Lifshay from comment #67)
> otherwise, you can start it manually:
> wg-quick up wg0
> 
> or look up how wg-quick works with sysv init.

I found a script that should work for sysv init:
https://www.procustodibus.com/blog/2021/06/wireguard-sysv-init-script/#with-sysv-init
Comment 70 Jacob Lifshay 2022-06-10 10:23:45 BST
I decided it'd probably be better for me to just run my own server that would be at build.libre-soc.org rather than using the libre-soc.org server, that way luke and alain wouldn't be blocking progress as much, they're busy enough with their own problems already.

after privately talking to luke a bit, he said that would help.
Comment 71 Jacob Lifshay 2022-06-10 22:18:38 BST
Luke, I'm getting the new server setup, can you add it to dns at build.libre-soc.org?

167.86.87.162
2a02:c207:2090:6249::1
Comment 72 Jacob Lifshay 2022-06-11 01:50:29 BST
(In reply to Jacob Lifshay from comment #71)
> Luke, I'm getting the new server setup, can you add it to dns at
> build.libre-soc.org?
> 
> 167.86.87.162
> 2a02:c207:2090:6249::1

I have http[s] all set up except for getting valid certificates, that's blocked on you setting up dns.
Comment 73 Jacob Lifshay 2022-06-14 04:21:33 BST
(In reply to Jacob Lifshay from comment #72)
> (In reply to Jacob Lifshay from comment #71)
> > Luke, I'm getting the new server setup, can you add it to dns at
> > build.libre-soc.org?
> > 
> > 167.86.87.162
> > 2a02:c207:2090:6249::1
> 
> I have http[s] all set up except for getting valid certificates, that's
> blocked on you setting up dns.

In order to make progress, I started setting it up to use build.libre-soc.programmerjake.tk instead...

I got git and https up and working. email is working but I need to get dmarc and dkim and spf working.
Comment 74 Luke Kenneth Casson Leighton 2022-06-14 14:04:57 BST
(In reply to Jacob Lifshay from comment #71)
> Luke, I'm getting the new server setup, can you add it to dns at
> build.libre-soc.org?
> 
> 167.86.87.162
> 2a02:c207:2090:6249::1

done, host gives this:

$ host build.libre-soc.org
build.libre-soc.org has address 167.86.87.162
build.libre-soc.org has IPv6 address 2a02:c207:2090:6249::1
Comment 75 Jacob Lifshay 2022-06-15 05:07:09 BST
(In reply to Luke Kenneth Casson Leighton from comment #74)
> (In reply to Jacob Lifshay from comment #71)
> > Luke, I'm getting the new server setup, can you add it to dns at
> > build.libre-soc.org?
> > 
> > 167.86.87.162
> > 2a02:c207:2090:6249::1
> 
> done, host gives this:
> 
> $ host build.libre-soc.org
> build.libre-soc.org has address 167.86.87.162
> build.libre-soc.org has IPv6 address 2a02:c207:2090:6249::1

ok. I got all the https certificates set up for build.libre-soc.org and got all the email and other stuff set up for build.libre-soc.programmerjake.tk set up.

I'm thinking it might be better to send all the emails and stuff from build.libre-soc.programmerjake.tk since that doesn't require your maintaining all the dns records and stuff.

I enabled sending build emails for lkcl@lkcl.net and for me. I want to add a unsubscribe link, as well as making a better way to subscribe to build emails than getting one of us to ssh into the server and edit
/home/build/gitlab-ci-archiver/allowed-emails.txt

Anyone else who wants to get build emails, let me/luke know... the system is WIP though, so expect oddities/issues.

Currently it just sends emails for build failures only to the tested commit's author and committer...any intervening commits don't get emails.
Comment 76 Jacob Lifshay 2022-06-16 09:52:19 BST
(In reply to Jacob Lifshay from comment #75)
> I enabled sending build emails for lkcl@lkcl.net and for me. I want to add a
> unsubscribe link, as well as making a better way to subscribe to build
> emails than getting one of us to ssh into the server and edit
> /home/build/gitlab-ci-archiver/allowed-emails.txt

I added code for unsubscribe/resubscribe, but didn't enable it yet...review of the CGI script appreciated!

https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/unsubscribe-cgi.sh

Library used by it:
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/allowed-emails.sh
Comment 77 Jacob Lifshay 2022-06-21 10:32:53 BST
(In reply to Jacob Lifshay from comment #76)
> I added code for unsubscribe/resubscribe, but didn't enable it yet...review
> of the CGI script appreciated!

After experiencing bash's propensity for inadvertently ignoring errors, which makes it particularly terrible for CGI scripts, I rewrote it in rust:
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/tree/master/email_manage

that link is to both the unsubscribe cgi binary and the email-manage binary (used for changing who gets emails for which projects from the command line, as well as generating urls to access the subscription editor). they're both actually symlinks to the same file like busybox.

I added per-project email subscriptions, so, for example, lkcl can be subscribed to nmutil but not soc since nmutil should pass CI so therefore he'd probably want to know when he broke it, but soc currently doesn't pass CI so the build failure messages are just annoying.

I added a second email verification step to avoid the issue where build log emails get posted to mailing lists and someone clicking on your unsubscribe link to try and fiddle with all the settings, now all they'll be able to do is send you unwanted email verification emails.

the verification emails are rate limited with exponential backoff, the default settings are you can send 2 emails immediately, then you have to wait 6hr, then 12hr, then 1 day, and so on till the link expires in 3 days. I figured that's a decent balance between not getting 7000 spam emails and actually being able to send yourself another verification email when necessary.

I'll enable it on the build server in the morning when I can be awake in case anything breaks.
Comment 78 Jacob Lifshay 2022-06-22 03:40:54 BST
(In reply to Jacob Lifshay from comment #77)
> I'll enable it on the build server in the morning when I can be awake in
> case anything breaks.

I enabled it and added documentation of how it works to the README.md:
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/README.md

As far as I can tell, it works!
Comment 79 Jacob Lifshay 2022-07-05 02:28:40 BST
I'm considering this completed. If we want to add a nicer web-based git viewer, that can be done as part of a different task.
Comment 80 Jacob Lifshay 2022-07-05 02:45:01 BST
lkcl, do you think the budget for this should be increased?
I did put in quite a lot of work:
Code written for gitlab ci:
All of:
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver
https://git.libre-soc.org/?p=utils.git;a=shortlog;h=refs/heads/gitlab-build-utils
I also modified git-mirror to use for our purposes:
https://salsa.debian.org/Kazan-team/git-mirror