Bug 190 - Setup Gitlab CI Runner for Kazan on a computer
Summary: Setup Gitlab CI Runner for Kazan on a computer
Status: CONFIRMED
Alias: None
Product: Libre-SOC Website
Classification: Unclassified
Component: website (show other bugs)
Version: unspecified
Hardware: All All
: Highest blocker
Assignee: Jacob Lifshay
URL:
Depends on:
Blocks:
 
Reported: 2020-02-26 17:15 GMT by Cole Poirier
Modified: 2021-08-08 11:04 BST (History)
3 users (show)

See Also:
NLnet milestone: ---
total budget (EUR) for completion of task and all subtasks: 0
budget (EUR) for this task, excluding subtasks' budget: 0
parent task for budget allocation:
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Cole Poirier 2020-02-26 17:15:41 GMT
Task: Figure out how to set up GitLab CI runner on a computer:

> Jacob: "I have an old 8-core AMD FX computer with 20GB of ram and a small SSD
that I'm not currently using, I've been wanting to set it up as our
own CI runner for Kazan-team on Debian Salsa, since Debian apparently
doesn't have that many spare CI runners and since CI builds will
require building LLVM (once I re-enable it) which is a really long
process."

I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory, 
500 GB ssd, rx 570 4GB desktop that I use to do some 
work on the evenings and weekends. I’d be happy to set 
this up on there if you want to use your old rig for other 
Debian CI.

> Jacob: "I can install Debian on it and give you a user account if I can get
your public SSH key."

Sounds good. Should I email the key to you, or put it on here? Additionally, is all that is needed to run "ssh-keygen -o" and complete the interactive steps? (Following the process here: https://www.keycdn.com/support/create-ssh-key).

> Jacob: "You would have to connect to the server using SSH over Tor, since I
don't have a publicly accessible IP address."

>> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?"

Jacob and Luke, which option is preferable? SSH over tor or openvpn?

> Jacob: "One requirement is to have GitLab set up such that we have to give
people explicit permission before they can run anything on it, since
I'm running it on my home network and I don't want to deal with abuse."

So to be clear, we will be using ssh over (see above) as well as an explicit request-permission grant system, for the actual running of tests/submission of jobs? Perhaps this permissions functionality for running CI jobs already exists in gitlab? I will find out once I read the installation docs.

> Jacob: "We can check with Luke what he thinks about how much money should be
assigned to this task, since this is more of a nice-to-have rather
than a requirement. I'm thinking maybe EUR 100 at most, and if it
looks like it's going to take longer than that, we should just give
up, since it's not that important."

I am interested in doing this nice to have. I don’t think a budget should be assigned to it, I’m happy to do this to help out and learn.

Jacob: "Since this is more of a sys-admin job rather than a Rust programming
job, it may not be quite what you were hoping for. There will
definitely be Rust programming jobs later."

Perfectly okay, I'm here to help in any way I can. Looking forward to doing some rust when the time comes :-)

Jacob: "Some additional requirements: Docker should be used as the backend runner."

Does it have to be docker, or can we use podman? Am I correct in my understanding that podman is the linux container utility?

Jacob: "It should be set up to block outgoing connections from the runners to
any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If possible, can you build a bash script that will install and set everything up from a fresh install of Debian, that way, it will be easier to recover from a corrupted system, install it on more computers, and publicly document the setup. Installation instructions here: https://docs.gitlab.com/12.8/runner/install/".

Yes, setting this up as a bash script sounds like an excellent idea, will be great for when something goes bad. With regard to all of the security and network configuration, it will most likely take me many tries to get right, so I'll be posting here for feedback on my configuration from time to time. And I assume both you and Luke will do a review to make sure I've set everything up so we don't get abused.
Comment 1 Jacob Lifshay 2020-02-28 23:11:17 GMT
(In reply to Cole Poirier from email)
> I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory,
> 500 GB ssd, rx 570 4GB desktop that I use to do some
> work on the evenings and weekends. I’d be happy to set
> this up on there if you want to use your old rig for other
> Debian CI.

My old computer is just sitting unused, so I think we should use it since it won't disturb you while your working.
Comment 2 Luke Kenneth Casson Leighton 2020-02-28 23:20:48 GMT
cole, chuck debian 10 on it, we can find sonething to do with it, it's a pretty mental spec, twice as powerful as my  previous laptop.

if you can also install openvpn and sshd i can connect it to my vpn
Comment 3 Jacob Lifshay 2020-02-28 23:39:36 GMT
(In reply to Cole Poirier from comment #0)
> > Jacob: "I can install Debian on it and give you a user account if I can get
> your public SSH key."
> 
> Sounds good. Should I email the key to you, or put it on here?

email it to me. Make sure you send me the id_rsa.pub and NOT the id_rsa, since id_rsa is the private key which allows anyone who has it to impersonate you.

> Additionally,
> is all that is needed to run "ssh-keygen -o" and complete the interactive
> steps? (Following the process here:
> https://www.keycdn.com/support/create-ssh-key).

All you need is to run "ssh-keygen -o" and follow the prompts.
You should use a non-blank passphrase when you setup your SSH key, since that makes it harder for someone who somehow obtained your private key to impersonate you, since they then have to guess your passphrase before they can use it.

> 
> > Jacob: "You would have to connect to the server using SSH over Tor, since I
> don't have a publicly accessible IP address."
> 
> >> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?"
> 
> Jacob and Luke, which option is preferable? SSH over tor or openvpn?

OpenVPN is preferred, I just have to research how to set that up first.

> > Jacob: "One requirement is to have GitLab set up such that we have to give
> people explicit permission before they can run anything on it, since
> I'm running it on my home network and I don't want to deal with abuse."
> 
> So to be clear, we will be using ssh over (see above) as well as an explicit
> request-permission grant system, for the actual running of tests/submission
> of jobs? Perhaps this permissions functionality for running CI jobs already
> exists in gitlab? I will find out once I read the installation docs.

SSH is used for you to run commands on the computer in order to do initial setup. GitLab has a different system that will run automatically once it's set up.

The permission system is totally separate -- on Debian Salsa itself -- probably the Developer permission.

> > Jacob: "We can check with Luke what he thinks about how much money should be
> assigned to this task, since this is more of a nice-to-have rather
> than a requirement. I'm thinking maybe EUR 100 at most, and if it
> looks like it's going to take longer than that, we should just give
> up, since it's not that important."
> 
> I am interested in doing this nice to have. I don’t think a budget should be
> assigned to it, I’m happy to do this to help out and learn.

:)

> Jacob: "Since this is more of a sys-admin job rather than a Rust programming
> job, it may not be quite what you were hoping for. There will
> definitely be Rust programming jobs later."
> 
> Perfectly okay, I'm here to help in any way I can. Looking forward to doing
> some rust when the time comes :-)
> 
> Jacob: "Some additional requirements: Docker should be used as the backend
> runner."
> 
> Does it have to be docker, or can we use podman? Am I correct in my
> understanding that podman is the linux container utility?

I'm not familiar with podman, I just know that Docker is supported as a backend and is what I'm currently using to run CI jobs on GitHub Actions. Also, Docker has some degree of isolation to make it harder to intentionally/accidentally mess up the server whereas some of the other GitLab Runner backends may not.

> Jacob: "It should be set up to block outgoing connections from the runners to
> any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If
> possible, can you build a bash script that will install and set everything
> up from a fresh install of Debian, that way, it will be easier to recover
> from a corrupted system, install it on more computers, and publicly document
> the setup. Installation instructions here:
> https://docs.gitlab.com/12.8/runner/install/".
> 
> Yes, setting this up as a bash script sounds like an excellent idea, will be
> great for when something goes bad. With regard to all of the security and
> network configuration, it will most likely take me many tries to get right,
> so I'll be posting here for feedback on my configuration from time to time.
> And I assume both you and Luke will do a review to make sure I've set
> everything up so we don't get abused.

I will try my best :)
Comment 4 Luke Kenneth Casson Leighton 2020-02-29 00:40:35 GMT
(In reply to Jacob Lifshay from comment 

> > Jacob and Luke, which option is preferable? SSH over tor or openvpn?
> 
> OpenVPN is preferred, I just have to research how to set that up first.

it's a total pain in the neck.

so 13 years ago i automated it via a wget cgi-bin script that creates an entire new config on-demand on the server and allows a .tgz archive to be downloaded which can be unpacked on the client.

it takes 30 seconds to join a new client and be up and running in 45 seconds.

don't waste precious time on it is what i am saying.
Comment 5 Jacob Lifshay 2020-03-05 05:55:06 GMT
Got my new cheap case (thought I shouldn't just keep using a cardboard box) and finished assembling the build server.

Currently installing debian on it, after which I will fix the fan controls because it has a small fan that currently runs full-blast and I have it stuck in my bedroom.

Specs:
AMD FX-8300
20GB RAM
Geforce GT710
Gigabyte ga990-fxa-ud7 motherboard
120GB SATA SSD
430W power supply

110MB/s down 12MB/s up network
Comment 6 Jacob Lifshay 2020-03-05 09:12:02 GMT
(In reply to Jacob Lifshay from comment #5)
> Currently installing debian on it, after which I will fix the fan controls
> because it has a small fan that currently runs full-blast and I have it
> stuck in my bedroom.

got fancontrol all set up!

Can figure out openvpn stuff tomorrow.
Comment 7 Luke Kenneth Casson Leighton 2020-03-06 00:47:05 GMT
(In reply to Jacob Lifshay from comment #6)

> got fancontrol all set up!

excellent
 
> Can figure out openvpn stuff tomorrow.

okaaay, run "sftp -P922 jacob@libre-riscv.org" using one of the two ssh keys you sent me, and you'll find there's a jacob.tgz archive.  unpack it somewhere then copy the contents to /etc/openvpn

you should then be able to see http://10.6.0.1

you *might* not be able to ping 10.6.0.1, i don't know why.  annoying because it is my normal way of checking if the connection is up aaaand of course it was failing.  bizarrely, traceroute works greeeeat.  *sigh*
Comment 8 Jacob Lifshay 2020-03-06 02:25:35 GMT
I tried Luke's openvpn setup, it didn't work. I got tor working, so, Cole (and Luke), I'm ready for you to send me your SSH public key (by private email).

To use it (from Debian):
sudo apt install tor
sudo service tor start
torsocks ssh <your-user-name>@<my-address.onion>

I'll send you the correct username and address once I setup your account (if the address isn't public, it's harder to attack).

Install process was really simple:
sudo apt install tor
sudo nano /etc/tor/torrc

add/uncomment following 2 lines in "location-hidden services" section:
HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 22 127.0.0.1:22

restart service:
sudo service tor restart

inspect hidden service host name:
sudo less /var/lib/tor/hidden_service/hostname
Comment 9 Jacob Lifshay 2020-03-06 02:26:12 GMT
The above instructions are assuming that ssh is already installed and working
Comment 10 Luke Kenneth Casson Leighton 2020-03-06 09:37:31 GMT
sorry, jacob, i didn't make it clear: i can't use tor (or gpg except for
signing rather than encryption).  strong encryption for me is out.
Comment 11 Luke Kenneth Casson Leighton 2020-03-06 10:19:57 GMT
btw you and cole will be fine using tor.
Comment 12 Jacob Lifshay 2020-03-10 22:47:34 GMT
Forgot to mention earlier that we got it working where you can ssh to the build server through libre-riscv.org
Comment 13 Luke Kenneth Casson Leighton 2020-03-11 01:52:48 GMT
yes. sorry! v busy.  ssh key is all that's needed, we worked out how to automatically ssh tunnel over a VPN link so it looks like the public connection is to jacob's private machine when it's not :)

so build server is up. who is configuring it? whoever it is, send me ssh public key and preferred username.
Comment 14 Jacob Lifshay 2020-07-21 22:51:13 BST
status update: as far as I know, the gitlab build daemon is installed and connected to debian salsa.

the scripts I wrote to automatically mirror git commits from git.libre-soc.org work, however they are currently disabled because they push to salsa once a minute even if there aren't any new commits and that was causing problems for debian salsa.

We *really* should just add a post-receive hook to the repos in git.libre-soc.org to automatically push to debian salsa instead of using the scripts I wrote which depend on polling once per minute. We can add libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing libre-soc to push without requiring a user account on debian salsa.

The Rust program to copy the build logs from debian salsa to a git repo works but is not deployed. the shell scripts that send out notification emails need to be written, they are ran by that rust program.

the build log mailing list needs to be setup.
Comment 15 Cole Poirier 2020-07-21 23:04:17 BST
(In reply to Jacob Lifshay from comment #14)
> status update: as far as I know, the gitlab build daemon is installed and
> connected to debian salsa.
> 
> the scripts I wrote to automatically mirror git commits from
> git.libre-soc.org work, however they are currently disabled because they
> push to salsa once a minute even if there aren't any new commits and that
> was causing problems for debian salsa.
> 
> We *really* should just add a post-receive hook to the repos in
> git.libre-soc.org to automatically push to debian salsa instead of using the
> scripts I wrote which depend on polling once per minute. We can add
> libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing
> libre-soc to push without requiring a user account on debian salsa.
> 
> The Rust program to copy the build logs from debian salsa to a git repo
> works but is not deployed. the shell scripts that send out notification
> emails need to be written, they are ran by that rust program.
> 
> the build log mailing list needs to be setup.

Thanks Jacob, that's very helpful. Can you break this down into a list of tasks? i.e. what needs to be done first and how you recommend I should do it? I would do this myself, but I don't have your knowledge.
Comment 16 Jacob Lifshay 2020-07-21 23:22:35 BST
(In reply to Cole Poirier from comment #15)
> Thanks Jacob, that's very helpful. Can you break this down into a list of
> tasks? i.e. what needs to be done first and how you recommend I should do
> it? I would do this myself, but I don't have your knowledge.

1. Add post-receive git hooks to git.libre-soc.org that mirror repos to debian salsa.

2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
create config.toml file based on
https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-config.toml

and change the email.command entry to something like:
[email]
command = ["/path/to/send-email.sh"]

and create a shell script send-email.sh containing the commands to create and send the emails as required. All of the values it needs to know are passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the build log is passed in on the standard input -- so you'd probably want to run `tail -n 50` to get the last 50 lines of the build log to include in the email. the full build log is included in the git commit to the build archiving repo, so you don't need to save all of it.
Comment 17 Jacob Lifshay 2020-07-21 23:24:02 BST
you (or more likely Luke or Alain) would setup the mailing list as part of step 2
Comment 18 Cole Poirier 2020-07-21 23:27:40 BST
(In reply to Jacob Lifshay from comment #17)
> you (or more likely Luke or Alain) would setup the mailing list as part of
> step 2

That's great, thanks for taking the time to lay it out so nicely for me Jacob. Hmm I guess it would be ci.libre-soc.org?
Comment 19 Jacob Lifshay 2020-07-21 23:35:32 BST
(In reply to Cole Poirier from comment #18)
> (In reply to Jacob Lifshay from comment #17)
> > you (or more likely Luke or Alain) would setup the mailing list as part of
> > step 2
> 
> That's great, thanks for taking the time to lay it out so nicely for me
> Jacob. Hmm I guess it would be ci.libre-soc.org?

I would expect the mailing list to be something like libre-soc-builds@lists.libre-soc.org

The CI server is jacob-build-server.programmerjake.tk, which is not publicly accessible.
Comment 20 Cole Poirier 2020-07-23 19:40:36 BST
(In reply to Jacob Lifshay from comment #16)
> 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to
> debian salsa.

Would this just be a bash file with the path .git/hooks/post-receive on the libre-soc git server with the following contents?

```
#!/bin/bash
git push salsa -f --mirror
```
Comment 21 Luke Kenneth Casson Leighton 2020-07-23 19:49:35 BST
(In reply to Cole Poirier from comment #20)
> (In reply to Jacob Lifshay from comment #16)
> > 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to
> > debian salsa.
> 
> Would this just be a bash file with the path .git/hooks/post-receive on the
> libre-soc git server with the following contents?

given that i often have a commit rate of 10 to 30 commits per day this still results in too large a load on debian's servers.

it would be better pushed to the other machine behind the VPN.
Comment 22 Jacob Lifshay 2020-07-23 19:55:17 BST
(In reply to Luke Kenneth Casson Leighton from comment #21)
> given that i often have a commit rate of 10 to 30 commits per day this still
> results in too large a load on debian's servers.

I think you may be overestimating the load -- remember we're using my server to do the actual CI builds, debian salsa is just coordinating. This also has the benefit that people can view the source on the mirror if they prefer gitlab's interface.
Comment 23 Luke Kenneth Casson Leighton 2020-07-23 19:58:18 BST
(In reply to Jacob Lifshay from comment #22)

> I think you may be overestimating the load -- remember we're using my server
> to do the actual CI builds,

ah great, i didn't realise.
Comment 24 Cole Poirier 2020-07-23 20:17:55 BST
(In reply to Jacob Lifshay from comment #22)
> (In reply to Luke Kenneth Casson Leighton from comment #21)
> > given that i often have a commit rate of 10 to 30 commits per day this still
> > results in too large a load on debian's servers.
> 
> I think you may be overestimating the load -- remember we're using my server
> to do the actual CI builds, debian salsa is just coordinating. This also has
> the benefit that people can view the source on the mirror if they prefer
> gitlab's interface.

Is my solution correct and I should move on to working on the email part, and you either Luke or Jacob will add this file to the libre-soc git repo?
Comment 25 Jacob Lifshay 2020-07-23 20:21:11 BST
(In reply to Cole Poirier from comment #24)
> Is my solution correct and I should move on to working on the email part,
> and you either Luke or Jacob will add this file to the libre-soc git repo?

yes, as far as I know. Luke or someone else would have to add it since I don't have access to the server, otherwise I would have added it myself a while ago.
Comment 26 Cole Poirier 2020-07-24 23:57:46 BST
(In reply to Jacob Lifshay from comment #16)
> 2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
> create config.toml file based on
> https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-
> config.toml
> 
> and change the email.command entry to something like:
> [email]
> command = ["/path/to/send-email.sh"]
> 
> and create a shell script send-email.sh containing the commands to create
> and send the emails as required. All of the values it needs to know are
> passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the
> build log is passed in on the standard input -- so you'd probably want to
> run `tail -n 50` to get the last 50 lines of the build log to include in the
> email. the full build log is included in the git commit to the build
> archiving repo, so you don't need to save all of it.

Given that this deals with the mailing list Luke, would this be a task that should be done by Alain?
Comment 27 Cole Poirier 2020-08-27 01:36:56 BST
(In reply to Cole Poirier from comment #26)
> (In reply to Jacob Lifshay from comment #16)
> > 2. Write email-sending script and set it up to be run by gitlab-ci-archiver:
> > create config.toml file based on
> > https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-
> > config.toml
> > 
> > and change the email.command entry to something like:
> > [email]
> > command = ["/path/to/send-email.sh"]
> > 
> > and create a shell script send-email.sh containing the commands to create
> > and send the emails as required. All of the values it needs to know are
> > passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the
> > build log is passed in on the standard input -- so you'd probably want to
> > run `tail -n 50` to get the last 50 lines of the build log to include in the
> > email. the full build log is included in the git commit to the build
> > archiving repo, so you don't need to save all of it.
> 
> Given that this deals with the mailing list Luke, would this be a task that
> should be done by Alain?

Just spoke to Jacob and he said that we can't use his build server/CI fully yet because the things he mentioned that need to be changed on our server haven't been completed yet. Luke can you remind Alain to do this along with the ISA manual latex work, and the other stuff he is doing?
Comment 28 Luke Kenneth Casson Leighton 2021-04-30 13:17:24 BST
(In reply to Jacob Lifshay from comment #19)

> I would expect the mailing list to be something like
> libre-soc-builds@lists.libre-soc.org

ah.  a mailing list.  i was expecting a simple redirector.

a mailing list will completely overwhelm the VM allocation within months, with what is effectively autogenerated output.

we cannot do that.

can you please instead set it up locally, and i will ask Alain to provide http proxy redirection.

if this was not autogenerated (reproducible) output i would not suggest this.

if bugs are noticed the output from a given run can be captured along with the git tags and a bugreport created, manually.
Comment 29 Jacob Lifshay 2021-04-30 13:53:49 BST
(In reply to Luke Kenneth Casson Leighton from comment #28)
> (In reply to Jacob Lifshay from comment #19)
> 
> > I would expect the mailing list to be something like
> > libre-soc-builds@lists.libre-soc.org
> 
> ah.  a mailing list.  i was expecting a simple redirector.
> 
> a mailing list will completely overwhelm the VM allocation within months,
> with what is effectively autogenerated output.

I'd be very surprised if that were the case, since I expect the emails to be on the order of 1-2kB each since they only contain ~60 lines of text. To use 1GB of space would take like 1,000,000 commits!

The actual build log is stored in a git repo where git is very good about compressing out redundancies across files (when objects get packed into a pack file).
Comment 30 Luke Kenneth Casson Leighton 2021-04-30 13:59:21 BST
(In reply to Jacob Lifshay from comment #29)

> I'd be very surprised if that were the case, since I expect the emails to be
> on the order of 1-2kB each since they only contain ~60 lines of text.

ok so it doesn't send the actual logs themselves as attachments.
then that's perfectly doable.
Comment 31 Jacob Lifshay 2021-04-30 14:04:23 BST
(In reply to Luke Kenneth Casson Leighton from comment #30)
> (In reply to Jacob Lifshay from comment #29)
> 
> > I'd be very surprised if that were the case, since I expect the emails to be
> > on the order of 1-2kB each since they only contain ~60 lines of text.
> 
> ok so it doesn't send the actual logs themselves as attachments.
> then that's perfectly doable.

Yup:
(comment #16)
> so you'd probably want to run `tail -n 50` to get the last 50 lines of the
> build log to include in the email. the full build log is included in the git
> commit to the build archiving repo, so you don't need to save all of it.
Comment 32 Luke Kenneth Casson Leighton 2021-08-06 20:34:16 BST
i've been thinking about this for some time, wondering what it is that makes me reticentvto action it, and i don't feel comfortable hosting large amounts of automated output (and having to back it up, and pay for the hosting, and ask Alain to host a duplicate backup).

instead please install a mailing list locally. mailman2 is trivial to set up.

if there are important bugs found the report and repro conditions may be duplicated and an appropriate bugreport raised where *ONLY* the relevant and crucial information may be recorded for aufit purposes.

even hosting "small" success reports is not useful information

closing as invalid.
Comment 33 Luke Kenneth Casson Leighton 2021-08-06 20:35:22 BST
ah this is not the mailing list one. or, it is, but the action is no longer Alain or myself.  removing Alain from the action list, a libre-soc list is no longer required.
Comment 34 Jacob Lifshay 2021-08-08 06:11:49 BST
email is still required to notify those who pushed a commit that broke the tests.
Comment 35 Luke Kenneth Casson Leighton 2021-08-08 11:04:15 BST
(In reply to Jacob Lifshay from comment #34)
> email is still required to notify those who pushed a commit that broke the
> tests.

which can be set up locally on the machine, which is part of the VPN,
and can use the local IP address of libre-soc.org 10.6.0.1 as an SMTP
relay.  installing ssmtpd would do the trick, it's extremely simple.