Task: Figure out how to set up GitLab CI runner on a computer: > Jacob: "I have an old 8-core AMD FX computer with 20GB of ram and a small SSD that I'm not currently using, I've been wanting to set it up as our own CI runner for Kazan-team on Debian Salsa, since Debian apparently doesn't have that many spare CI runners and since CI builds will require building LLVM (once I re-enable it) which is a really long process." I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory, 500 GB ssd, rx 570 4GB desktop that I use to do some work on the evenings and weekends. I’d be happy to set this up on there if you want to use your old rig for other Debian CI. > Jacob: "I can install Debian on it and give you a user account if I can get your public SSH key." Sounds good. Should I email the key to you, or put it on here? Additionally, is all that is needed to run "ssh-keygen -o" and complete the interactive steps? (Following the process here: https://www.keycdn.com/support/create-ssh-key). > Jacob: "You would have to connect to the server using SSH over Tor, since I don't have a publicly accessible IP address." >> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?" Jacob and Luke, which option is preferable? SSH over tor or openvpn? > Jacob: "One requirement is to have GitLab set up such that we have to give people explicit permission before they can run anything on it, since I'm running it on my home network and I don't want to deal with abuse." So to be clear, we will be using ssh over (see above) as well as an explicit request-permission grant system, for the actual running of tests/submission of jobs? Perhaps this permissions functionality for running CI jobs already exists in gitlab? I will find out once I read the installation docs. > Jacob: "We can check with Luke what he thinks about how much money should be assigned to this task, since this is more of a nice-to-have rather than a requirement. I'm thinking maybe EUR 100 at most, and if it looks like it's going to take longer than that, we should just give up, since it's not that important." I am interested in doing this nice to have. I don’t think a budget should be assigned to it, I’m happy to do this to help out and learn. Jacob: "Since this is more of a sys-admin job rather than a Rust programming job, it may not be quite what you were hoping for. There will definitely be Rust programming jobs later." Perfectly okay, I'm here to help in any way I can. Looking forward to doing some rust when the time comes :-) Jacob: "Some additional requirements: Docker should be used as the backend runner." Does it have to be docker, or can we use podman? Am I correct in my understanding that podman is the linux container utility? Jacob: "It should be set up to block outgoing connections from the runners to any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If possible, can you build a bash script that will install and set everything up from a fresh install of Debian, that way, it will be easier to recover from a corrupted system, install it on more computers, and publicly document the setup. Installation instructions here: https://docs.gitlab.com/12.8/runner/install/". Yes, setting this up as a bash script sounds like an excellent idea, will be great for when something goes bad. With regard to all of the security and network configuration, it will most likely take me many tries to get right, so I'll be posting here for feedback on my configuration from time to time. And I assume both you and Luke will do a review to make sure I've set everything up so we don't get abused.
(In reply to Cole Poirier from email) > I have a Ryzen 2600X 6 core, 12 thread, 32 GB memory, > 500 GB ssd, rx 570 4GB desktop that I use to do some > work on the evenings and weekends. I’d be happy to set > this up on there if you want to use your old rig for other > Debian CI. My old computer is just sitting unused, so I think we should use it since it won't disturb you while your working.
cole, chuck debian 10 on it, we can find sonething to do with it, it's a pretty mental spec, twice as powerful as my previous laptop. if you can also install openvpn and sshd i can connect it to my vpn
(In reply to Cole Poirier from comment #0) > > Jacob: "I can install Debian on it and give you a user account if I can get > your public SSH key." > > Sounds good. Should I email the key to you, or put it on here? email it to me. Make sure you send me the id_rsa.pub and NOT the id_rsa, since id_rsa is the private key which allows anyone who has it to impersonate you. > Additionally, > is all that is needed to run "ssh-keygen -o" and complete the interactive > steps? (Following the process here: > https://www.keycdn.com/support/create-ssh-key). All you need is to run "ssh-keygen -o" and follow the prompts. You should use a non-blank passphrase when you setup your SSH key, since that makes it harder for someone who somehow obtained your private key to impersonate you, since they then have to guess your passphrase before they can use it. > > > Jacob: "You would have to connect to the server using SSH over Tor, since I > don't have a publicly accessible IP address." > > >> Luke: "I have openvpn installed and a braindead script for manually joining new devices. I can set that up very quickly, it may be easier than tor?" > > Jacob and Luke, which option is preferable? SSH over tor or openvpn? OpenVPN is preferred, I just have to research how to set that up first. > > Jacob: "One requirement is to have GitLab set up such that we have to give > people explicit permission before they can run anything on it, since > I'm running it on my home network and I don't want to deal with abuse." > > So to be clear, we will be using ssh over (see above) as well as an explicit > request-permission grant system, for the actual running of tests/submission > of jobs? Perhaps this permissions functionality for running CI jobs already > exists in gitlab? I will find out once I read the installation docs. SSH is used for you to run commands on the computer in order to do initial setup. GitLab has a different system that will run automatically once it's set up. The permission system is totally separate -- on Debian Salsa itself -- probably the Developer permission. > > Jacob: "We can check with Luke what he thinks about how much money should be > assigned to this task, since this is more of a nice-to-have rather > than a requirement. I'm thinking maybe EUR 100 at most, and if it > looks like it's going to take longer than that, we should just give > up, since it's not that important." > > I am interested in doing this nice to have. I don’t think a budget should be > assigned to it, I’m happy to do this to help out and learn. :) > Jacob: "Since this is more of a sys-admin job rather than a Rust programming > job, it may not be quite what you were hoping for. There will > definitely be Rust programming jobs later." > > Perfectly okay, I'm here to help in any way I can. Looking forward to doing > some rust when the time comes :-) > > Jacob: "Some additional requirements: Docker should be used as the backend > runner." > > Does it have to be docker, or can we use podman? Am I correct in my > understanding that podman is the linux container utility? I'm not familiar with podman, I just know that Docker is supported as a backend and is what I'm currently using to run CI jobs on GitHub Actions. Also, Docker has some degree of isolation to make it harder to intentionally/accidentally mess up the server whereas some of the other GitLab Runner backends may not. > Jacob: "It should be set up to block outgoing connections from the runners to > any local addresses (192.168.x.x, 10.x.x.x, etc.) Don't forget IPv6. If > possible, can you build a bash script that will install and set everything > up from a fresh install of Debian, that way, it will be easier to recover > from a corrupted system, install it on more computers, and publicly document > the setup. Installation instructions here: > https://docs.gitlab.com/12.8/runner/install/". > > Yes, setting this up as a bash script sounds like an excellent idea, will be > great for when something goes bad. With regard to all of the security and > network configuration, it will most likely take me many tries to get right, > so I'll be posting here for feedback on my configuration from time to time. > And I assume both you and Luke will do a review to make sure I've set > everything up so we don't get abused. I will try my best :)
(In reply to Jacob Lifshay from comment > > Jacob and Luke, which option is preferable? SSH over tor or openvpn? > > OpenVPN is preferred, I just have to research how to set that up first. it's a total pain in the neck. so 13 years ago i automated it via a wget cgi-bin script that creates an entire new config on-demand on the server and allows a .tgz archive to be downloaded which can be unpacked on the client. it takes 30 seconds to join a new client and be up and running in 45 seconds. don't waste precious time on it is what i am saying.
Got my new cheap case (thought I shouldn't just keep using a cardboard box) and finished assembling the build server. Currently installing debian on it, after which I will fix the fan controls because it has a small fan that currently runs full-blast and I have it stuck in my bedroom. Specs: AMD FX-8300 20GB RAM Geforce GT710 Gigabyte ga990-fxa-ud7 motherboard 120GB SATA SSD 430W power supply 110MB/s down 12MB/s up network
(In reply to Jacob Lifshay from comment #5) > Currently installing debian on it, after which I will fix the fan controls > because it has a small fan that currently runs full-blast and I have it > stuck in my bedroom. got fancontrol all set up! Can figure out openvpn stuff tomorrow.
(In reply to Jacob Lifshay from comment #6) > got fancontrol all set up! excellent > Can figure out openvpn stuff tomorrow. okaaay, run "sftp -P922 jacob@libre-riscv.org" using one of the two ssh keys you sent me, and you'll find there's a jacob.tgz archive. unpack it somewhere then copy the contents to /etc/openvpn you should then be able to see http://10.6.0.1 you *might* not be able to ping 10.6.0.1, i don't know why. annoying because it is my normal way of checking if the connection is up aaaand of course it was failing. bizarrely, traceroute works greeeeat. *sigh*
I tried Luke's openvpn setup, it didn't work. I got tor working, so, Cole (and Luke), I'm ready for you to send me your SSH public key (by private email). To use it (from Debian): sudo apt install tor sudo service tor start torsocks ssh <your-user-name>@<my-address.onion> I'll send you the correct username and address once I setup your account (if the address isn't public, it's harder to attack). Install process was really simple: sudo apt install tor sudo nano /etc/tor/torrc add/uncomment following 2 lines in "location-hidden services" section: HiddenServiceDir /var/lib/tor/hidden_service/ HiddenServicePort 22 127.0.0.1:22 restart service: sudo service tor restart inspect hidden service host name: sudo less /var/lib/tor/hidden_service/hostname
The above instructions are assuming that ssh is already installed and working
sorry, jacob, i didn't make it clear: i can't use tor (or gpg except for signing rather than encryption). strong encryption for me is out.
btw you and cole will be fine using tor.
Forgot to mention earlier that we got it working where you can ssh to the build server through libre-riscv.org
yes. sorry! v busy. ssh key is all that's needed, we worked out how to automatically ssh tunnel over a VPN link so it looks like the public connection is to jacob's private machine when it's not :) so build server is up. who is configuring it? whoever it is, send me ssh public key and preferred username.
status update: as far as I know, the gitlab build daemon is installed and connected to debian salsa. the scripts I wrote to automatically mirror git commits from git.libre-soc.org work, however they are currently disabled because they push to salsa once a minute even if there aren't any new commits and that was causing problems for debian salsa. We *really* should just add a post-receive hook to the repos in git.libre-soc.org to automatically push to debian salsa instead of using the scripts I wrote which depend on polling once per minute. We can add libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing libre-soc to push without requiring a user account on debian salsa. The Rust program to copy the build logs from debian salsa to a git repo works but is not deployed. the shell scripts that send out notification emails need to be written, they are ran by that rust program. the build log mailing list needs to be setup.
(In reply to Jacob Lifshay from comment #14) > status update: as far as I know, the gitlab build daemon is installed and > connected to debian salsa. > > the scripts I wrote to automatically mirror git commits from > git.libre-soc.org work, however they are currently disabled because they > push to salsa once a minute even if there aren't any new commits and that > was causing problems for debian salsa. > > We *really* should just add a post-receive hook to the repos in > git.libre-soc.org to automatically push to debian salsa instead of using the > scripts I wrote which depend on polling once per minute. We can add > libre-soc's ssh key to the repos on debian salsa as deploy keys, allowing > libre-soc to push without requiring a user account on debian salsa. > > The Rust program to copy the build logs from debian salsa to a git repo > works but is not deployed. the shell scripts that send out notification > emails need to be written, they are ran by that rust program. > > the build log mailing list needs to be setup. Thanks Jacob, that's very helpful. Can you break this down into a list of tasks? i.e. what needs to be done first and how you recommend I should do it? I would do this myself, but I don't have your knowledge.
(In reply to Cole Poirier from comment #15) > Thanks Jacob, that's very helpful. Can you break this down into a list of > tasks? i.e. what needs to be done first and how you recommend I should do > it? I would do this myself, but I don't have your knowledge. 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to debian salsa. 2. Write email-sending script and set it up to be run by gitlab-ci-archiver: create config.toml file based on https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example-config.toml and change the email.command entry to something like: [email] command = ["/path/to/send-email.sh"] and create a shell script send-email.sh containing the commands to create and send the emails as required. All of the values it needs to know are passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the build log is passed in on the standard input -- so you'd probably want to run `tail -n 50` to get the last 50 lines of the build log to include in the email. the full build log is included in the git commit to the build archiving repo, so you don't need to save all of it.
you (or more likely Luke or Alain) would setup the mailing list as part of step 2
(In reply to Jacob Lifshay from comment #17) > you (or more likely Luke or Alain) would setup the mailing list as part of > step 2 That's great, thanks for taking the time to lay it out so nicely for me Jacob. Hmm I guess it would be ci.libre-soc.org?
(In reply to Cole Poirier from comment #18) > (In reply to Jacob Lifshay from comment #17) > > you (or more likely Luke or Alain) would setup the mailing list as part of > > step 2 > > That's great, thanks for taking the time to lay it out so nicely for me > Jacob. Hmm I guess it would be ci.libre-soc.org? I would expect the mailing list to be something like libre-soc-builds@lists.libre-soc.org The CI server is jacob-build-server.programmerjake.tk, which is not publicly accessible.
(In reply to Jacob Lifshay from comment #16) > 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to > debian salsa. Would this just be a bash file with the path .git/hooks/post-receive on the libre-soc git server with the following contents? ``` #!/bin/bash git push salsa -f --mirror ```
(In reply to Cole Poirier from comment #20) > (In reply to Jacob Lifshay from comment #16) > > 1. Add post-receive git hooks to git.libre-soc.org that mirror repos to > > debian salsa. > > Would this just be a bash file with the path .git/hooks/post-receive on the > libre-soc git server with the following contents? given that i often have a commit rate of 10 to 30 commits per day this still results in too large a load on debian's servers. it would be better pushed to the other machine behind the VPN.
(In reply to Luke Kenneth Casson Leighton from comment #21) > given that i often have a commit rate of 10 to 30 commits per day this still > results in too large a load on debian's servers. I think you may be overestimating the load -- remember we're using my server to do the actual CI builds, debian salsa is just coordinating. This also has the benefit that people can view the source on the mirror if they prefer gitlab's interface.
(In reply to Jacob Lifshay from comment #22) > I think you may be overestimating the load -- remember we're using my server > to do the actual CI builds, ah great, i didn't realise.
(In reply to Jacob Lifshay from comment #22) > (In reply to Luke Kenneth Casson Leighton from comment #21) > > given that i often have a commit rate of 10 to 30 commits per day this still > > results in too large a load on debian's servers. > > I think you may be overestimating the load -- remember we're using my server > to do the actual CI builds, debian salsa is just coordinating. This also has > the benefit that people can view the source on the mirror if they prefer > gitlab's interface. Is my solution correct and I should move on to working on the email part, and you either Luke or Jacob will add this file to the libre-soc git repo?
(In reply to Cole Poirier from comment #24) > Is my solution correct and I should move on to working on the email part, > and you either Luke or Jacob will add this file to the libre-soc git repo? yes, as far as I know. Luke or someone else would have to add it since I don't have access to the server, otherwise I would have added it myself a while ago.
(In reply to Jacob Lifshay from comment #16) > 2. Write email-sending script and set it up to be run by gitlab-ci-archiver: > create config.toml file based on > https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example- > config.toml > > and change the email.command entry to something like: > [email] > command = ["/path/to/send-email.sh"] > > and create a shell script send-email.sh containing the commands to create > and send the emails as required. All of the values it needs to know are > passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the > build log is passed in on the standard input -- so you'd probably want to > run `tail -n 50` to get the last 50 lines of the build log to include in the > email. the full build log is included in the git commit to the build > archiving repo, so you don't need to save all of it. Given that this deals with the mailing list Luke, would this be a task that should be done by Alain?
(In reply to Cole Poirier from comment #26) > (In reply to Jacob Lifshay from comment #16) > > 2. Write email-sending script and set it up to be run by gitlab-ci-archiver: > > create config.toml file based on > > https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/example- > > config.toml > > > > and change the email.command entry to something like: > > [email] > > command = ["/path/to/send-email.sh"] > > > > and create a shell script send-email.sh containing the commands to create > > and send the emails as required. All of the values it needs to know are > > passed in using environment variables starting with GITLAB_CI_ARCHIVER_; the > > build log is passed in on the standard input -- so you'd probably want to > > run `tail -n 50` to get the last 50 lines of the build log to include in the > > email. the full build log is included in the git commit to the build > > archiving repo, so you don't need to save all of it. > > Given that this deals with the mailing list Luke, would this be a task that > should be done by Alain? Just spoke to Jacob and he said that we can't use his build server/CI fully yet because the things he mentioned that need to be changed on our server haven't been completed yet. Luke can you remind Alain to do this along with the ISA manual latex work, and the other stuff he is doing?
(In reply to Jacob Lifshay from comment #19) > I would expect the mailing list to be something like > libre-soc-builds@lists.libre-soc.org ah. a mailing list. i was expecting a simple redirector. a mailing list will completely overwhelm the VM allocation within months, with what is effectively autogenerated output. we cannot do that. can you please instead set it up locally, and i will ask Alain to provide http proxy redirection. if this was not autogenerated (reproducible) output i would not suggest this. if bugs are noticed the output from a given run can be captured along with the git tags and a bugreport created, manually.
(In reply to Luke Kenneth Casson Leighton from comment #28) > (In reply to Jacob Lifshay from comment #19) > > > I would expect the mailing list to be something like > > libre-soc-builds@lists.libre-soc.org > > ah. a mailing list. i was expecting a simple redirector. > > a mailing list will completely overwhelm the VM allocation within months, > with what is effectively autogenerated output. I'd be very surprised if that were the case, since I expect the emails to be on the order of 1-2kB each since they only contain ~60 lines of text. To use 1GB of space would take like 1,000,000 commits! The actual build log is stored in a git repo where git is very good about compressing out redundancies across files (when objects get packed into a pack file).
(In reply to Jacob Lifshay from comment #29) > I'd be very surprised if that were the case, since I expect the emails to be > on the order of 1-2kB each since they only contain ~60 lines of text. ok so it doesn't send the actual logs themselves as attachments. then that's perfectly doable.
(In reply to Luke Kenneth Casson Leighton from comment #30) > (In reply to Jacob Lifshay from comment #29) > > > I'd be very surprised if that were the case, since I expect the emails to be > > on the order of 1-2kB each since they only contain ~60 lines of text. > > ok so it doesn't send the actual logs themselves as attachments. > then that's perfectly doable. Yup: (comment #16) > so you'd probably want to run `tail -n 50` to get the last 50 lines of the > build log to include in the email. the full build log is included in the git > commit to the build archiving repo, so you don't need to save all of it.
i've been thinking about this for some time, wondering what it is that makes me reticentvto action it, and i don't feel comfortable hosting large amounts of automated output (and having to back it up, and pay for the hosting, and ask Alain to host a duplicate backup). instead please install a mailing list locally. mailman2 is trivial to set up. if there are important bugs found the report and repro conditions may be duplicated and an appropriate bugreport raised where *ONLY* the relevant and crucial information may be recorded for aufit purposes. even hosting "small" success reports is not useful information closing as invalid.
ah this is not the mailing list one. or, it is, but the action is no longer Alain or myself. removing Alain from the action list, a libre-soc list is no longer required.
email is still required to notify those who pushed a commit that broke the tests.
(In reply to Jacob Lifshay from comment #34) > email is still required to notify those who pushed a commit that broke the > tests. which can be set up locally on the machine, which is part of the VPN, and can use the local IP address of libre-soc.org 10.6.0.1 as an SMTP relay. installing ssmtpd would do the trick, it's extremely simple.
started working on this again after some prompting: https://libre-soc.org/irclog/%23libre-soc.2022-03-29.log.html#t2022-03-29T08:46:00 The old script I wrote to do git mirroring didn't work very well -- the Salsa admins told me to turn it off because it was abusing their server by pushing nothing every 1 or 5min (icr which). In order to fix that, I modified the code from https://git.sr.ht/~janbaudisch/git-mirror to only push to the target repo if there were changes. I also added debian packaging and modified it to use more standard directories for debian. it runs from a systemd timer. I also added tests. Modified repo: https://salsa.debian.org/Kazan-team/git-mirror git has no good mechanism that I could find to tell you if `git fetch` did anything, so I just set it to create 2 local repos and push from one to the other (using the ext protocol to call git-receive-pack -- avoiding needing to go through ssh or some other internet thing). The target repo has a post-receive hook that pushes to the final remote repo, since post-receive is only called when things change. (all of this would be avoided if luke would just install a post-receive hook on git.libre-soc.org, but he said no iirc) I'll install it on the build server after getting some sleep.
i forgot to set git-mirror to run as non-root (handy for having a separate ssh key, as well as security), so I'll add that next.
I got the mirrors all set up and running, the mirror scripts needed a bunch of misc. fixes...good thing I have tests now! I moved all the mirror repos to the mirrors subgroup: https://salsa.debian.org/Kazan-team/mirrors
I updated gitlab-ci-archiver and wrote an email-sending script, I'll try installing it tomorrow. https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/send-email.sh I also set up the postfix server and tested it, couldn't get it to work, afaict libre-soc.org's mail server isn't configured to let my server send anything. Luke, can you try to take care of that if you have time? I may be able to send some test emails in an hour or so if you like. (or you can log into my build server and send some yourself, using /usr/sbin/sendmail)
$ telnet 10.6.0.1 25 Trying 10.6.0.1... Connected to 10.6.0.1. Escape character is '^]'. 421 Too many concurrent SMTP connections; please try again later. step 1, check that's ok step 2, reverse-dns (now added 10.6.0.7) step 3, configure the sender correctly. i said "make it look like libre-soc.org". you've made it look like "build1.libre-soc.org" which fails sender-verify.
urr i had to actually set up reverse-dns in bind *shudder* try this from build1 to confirm: $ telnet 10.6.0.1 25 Trying 10.6.0.1... Connected to 10.6.0.1. Escape character is '^]'. 220 libre-soc.org ESMTP Exim 4.92 Thu, 31 Mar 2022 10:41:20 +0100 HELO localhost <---- TYPE 250 libre-soc.org Hello fizzy.vpn [10.6.0.5] MAIL FROM:lkcl@lkcl.net <---- TYPE 250 OK RCPT TO:luke.leighton@gmail.com <----- TYPE 250 Accepted DATA <------ TYPE 354 Enter message, ending with "." on a line by itself From: lkcl@lkcl.net <------ TYPE all below To: luke.leighton@gmail.com Subject: test test . 250 OK id=1nZrJD-0007yx-K8 and configure build1 to pretend to *ONLY* be libre-soc.org *NOT* repeat *NOT* build1.libre-soc.org
(In reply to Luke Kenneth Casson Leighton from comment #41) > and configure build1 to pretend to *ONLY* be libre-soc.org > *NOT* repeat *NOT* build1.libre-soc.org i changed postfix's config to try to be libre-soc.org...i sent you a bounce i got after that.
yep you sent it from jacob@libre-soc.org which is a non-existent email address, and you (legitimately) got a notification "sender verification failed" i've added jacob@libre-soc.org so that should now succeed.
Created attachment 160 [details] demo email i set gitlab-ci-archiver to work archiving all the previous pipelines, i disabled sending emails for that. (assuming there isn't any bugs) tomorrow i'll set up the cron job or systemd timer for running it and re-enable sending emails. any ideas for a good allow-list of who to send emails to? i want to avoid spamming random people if we import linux's source code or something. i'm thinking go through the list of people on our budget tracking software and use that as an initial email list. next, i'll need to make the git repo publicly visible for both https clones and gitweb (or something similar).
(In reply to Luke Kenneth Casson Leighton from comment #41) > urr i had to actually set up reverse-dns in bind *shudder* well, reverse-dns broke again: jacob@jacob-build-server:~$ { printf 'HELO jacob-build-server.programmerjake.tk\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25 220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:11:28 +0100 250 libre-soc.org Hello jacob-build-server.programmerjake.tk [10.6.0.7] 250 OK 550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed) 221 libre-soc.org closing connection jacob@jacob-build-server:~$ { printf 'HELO libre-soc.org\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25 220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:12:03 +0100 250 libre-soc.org Hello libre-soc.org [10.6.0.7] 250 OK 550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed) 221 libre-soc.org closing connection jacob@jacob-build-server:~$ { printf 'HELO localhost\r\n'; sleep 1; printf 'MAIL FROM:<build@libre-soc.org>\r\n'; sleep 1; printf 'RCPT TO:<programmerjake@gmail.com>\r\n'; sleep 10; printf "QUIT\r\n"; } | nc 10.6.0.1 25 220 libre-soc.org ESMTP Exim 4.92 Fri, 01 Apr 2022 04:13:16 +0100 250 libre-soc.org Hello localhost [10.6.0.7] 250 OK 550 X-Host-Lookup-Failed: Reverse DNS lookup failed for 10.6.0.7 (failed) 221 libre-soc.org closing connection
I got nginx set up on my build server and confirmed that I could proxy to it using nginx on another computer. I have an index.html for the root, and I have /build-archive.git pointed to the git repo (as a dumb git server, since that's easier to set up and since it's a totally static website). once that's set up, you can clone: git clone https://build.programmerjake.tk/build-archive.git needed nginx proxy config (modify as needed for libre-soc.org): server { listen 80; listen [::]:80; listen 443 ssl; listen [::]:443 ssl; server_name build.programmerjake.tk; ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem; # Redirect non-https traffic to https if ($scheme != "https") { return 301 https://$host$request_uri; } location / { proxy_pass http://jacob-build-server.programmerjake.tk/; } } Modifications needed: * build.programmerjake.tk will need to be replaced with build.libre-soc.org * ssl cert paths need to be adjusted * proxy_pass needs to have the domain name replaced with 10.6.0.7 Additional steps: * set up build.libre-soc.org to point to libre-soc.org's server * add a ssl certificate, or re-issue the existing one but with build.libre-soc.org added as a subject alternate name.
(In reply to Jacob Lifshay from comment #46) > * add a ssl certificate, or re-issue the existing one but with > build.libre-soc.org added as a subject alternate name. If you're using http[s] for letsencrypt's validation, the nginx server block will need adjusting to handle the /.well-known/acme-challenge path on libre-soc.org's server rather than proxying it.
proxying /.well-known: server { listen 80; listen [::]:80; listen 443 ssl; listen [::]:443 ssl; server_name build.programmerjake.tk; ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem; # Redirect non-https traffic to https if ($scheme != "https") { return 301 https://$host$request_uri; } location /.well-known { root /var/www/html; index index.html index.htm index.nginx-debian.html; try_files $uri $uri/ =404; } location / { proxy_pass http://jacob-build-server.programmerjake.tk/; } }
we need alain for that one because he created alternative scripts (for security reasons) for acme.
I now have an fpga to add to the CI server: https://bugs.libre-soc.org/show_bug.cgi?id=774#c21
tried passing through the tty devices, discovered that's far from sufficient: https://libre-soc.org/irclog/%23libre-soc.2022-04-16.log.html#t2022-04-16T00:56:51
(In reply to Jacob Lifshay from comment #51) > tried passing through the tty devices, discovered that's far from sufficient: > https://libre-soc.org/irclog/%23libre-soc.2022-04-16.log.html#t2022-04-16T00: > 56:51 i'll probably try luke's suggestion of running xc3sprog outside docker via ssh
did more research into passing the tty devices into docker so scripts can see the fpga's uart: tty devices are dynamically created and removed during the fpga programming process -- docker's method of dealing with dynamic devices: https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/man/src/container/create-example.md#dealing-with-dynamically-created-devices---device-cgroup-rule gitlab-runner merge request for adding the device cgroup rules option: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309 afaict gitlab-runner uses docker through the go library rather than through the docker executable, so making a wrapper script probably won't work
I got blinky to successfully program using gitlab-runner and docker on my desktop, I'll put it on the build server tomorrow. Took a while to get all the required stuff working (nextpnr-xilinx and all that fun stuff). I realized that ssh wouldn't really work since the docker container wouldn't have any ssh keys that were valid for the host, and I didn't really want it to have any...I realize that could be worked around, but it sounds like a pain. So instead, I wrote a python program that tunnels out of docker using a unix socket, the daemon outside runs xc3sprog (as well as saves a copy of the last .bit file, came in handy for testing). the unix socket can be put inside the docker container by using the gitlab-runner config to set up a docker volume that is just the unix socket file. apparently xc3sprog wants its stdout/stderr to be a terminal, so I had to figure that out too. I want to move program-fpga.py to another repo so we have a centralized location for it so we don't have a bunch of copies getting out of sync, does utils.git sound good? https://salsa.debian.org/Kazan-team/gitlab-runner-test/-/commit/0447eefda9e443842e09b11672d73082a3df83c6
(In reply to Jacob Lifshay from comment #54) > I want to move program-fpga.py to another repo so we have a centralized > location for it so we don't have a bunch of copies getting out of sync, does > utils.git sound good? I'll just move it to utils.git.
I got blinky.py to work, programming a fpga successfully when run in gitlab-ci: https://salsa.debian.org/Kazan-team/mirrors/utils/-/jobs/2685425#L7937 Other repos should add utils.git as a submodule on branch gitlab-build-utils and set XC3SPROG="$PWD/gitlab-build-utils/program-fpga.py" https://git.libre-soc.org/?p=utils.git;a=tree;h=d6a59e33c7c197d0c61abe7f4ae5666c95535b89;hb=d6a59e33c7c197d0c61abe7f4ae5666c95535b89 On the build server, the systemd service program-fpga.service handles running xc3sprog, the socket /run/program-fpga.sock is mounted at /arty-prog-socket inside the docker containers used by gitlab-ci.
(In reply to Jacob Lifshay from comment #56) > I got blinky.py to work, programming a fpga successfully when run in > gitlab-ci: > https://salsa.debian.org/Kazan-team/mirrors/utils/-/jobs/2685425#L7937 > > Other repos should add utils.git as a submodule on branch gitlab-build-utils > and set XC3SPROG="$PWD/gitlab-build-utils/program-fpga.py" > I added a readme with instructions (as well as instructions for installing it on the build server, in case we need them later): Rendered: https://salsa.debian.org/Kazan-team/mirrors/utils/-/blob/ec98be986b007dcac889d7b5077b962fe73b030a/README.md https://git.libre-soc.org/?p=utils.git;a=blob;f=README.md;h=c219e164330c02f5fcdfedf9b8abe4e578215d19;hb=ec98be986b007dcac889d7b5077b962fe73b030a
I added a uart_demo.py that programs the FPGA with a simple transmit-only UART that's connected to all UART TX pins defined by the platform and transmits some text in a repeating loop. The default text is "Hello World!\n" and the default baud rate is 9600. I built it with gitlab-ci and it successfully programmed the fpga, but since I don't yet have the serial port passed into the gitlab-ci docker container, I just logged onto the build server to test it...it works! https://git.libre-soc.org/?p=utils.git;a=blob;f=uart_demo.py;h=89c40e71fc544d16e4c02464b00f20fb3d95ad05;hb=289ae5cc49eee8380210e09db3a8ec3f12ca176a The reason I wrote the simple UART is so we have something to test that the FPGA on gitlab-ci works without needing to build a whole soft-cpu that could easily be broken by someone while their working on it...uart_demo.py is unlikely to break. I'll add code that reads from the serial port from inside the docker container and checks that it got the right text and didn't time-out...at that point, I can declare the FPGA to be officially working on gitlab-ci...I'll wire up the hyperram later.
(In reply to Jacob Lifshay from comment #58) > I added a uart_demo.py that programs the FPGA with a simple transmit-only > UART that's connected to all UART TX pins defined by the platform and > transmits some text in a repeating loop. The default text is "Hello > World!\n" and the default baud rate is 9600. ooo, that's worthwhile adding to nmigen in the same directory as blinky (in the same style: generic code separated from platform) > The reason I wrote the simple UART is so we have something to test that the > FPGA on gitlab-ci works without needing to build a whole soft-cpu that could > easily be broken by someone while their working on it... no great idea > uart_demo.py is unlikely to break. yeah totally get it. > I'll add code that reads from the serial port from inside the docker > container and checks that it got the right text and didn't time-out...at > that point, I can declare the FPGA to be officially working on > gitlab-ci...I'll wire up the hyperram later. components face-up.
(In reply to Luke Kenneth Casson Leighton from comment #49) > we need alain for that one because he created alternative > scripts (for security reasons) for acme. It's been a month and a half, since it doesn't seem like alain will get around to this anytime soon, lkcl can you please work on this?
no, i can't. use a relative URL instead.
(In reply to Luke Kenneth Casson Leighton from comment #61) > no, i can't. use a relative URL instead. hmm, another option is to have the tls on my end... https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on-hostname/40135151#40135151
(In reply to Jacob Lifshay from comment #62) > (In reply to Luke Kenneth Casson Leighton from comment #61) > > no, i can't. use a relative URL instead. > > hmm, another option is to have the tls on my end... > https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on- > hostname/40135151#40135151 lkcl on irc: https://libre-soc.org/irclog/%23libre-soc.2022-05-23.log.html#t2022-05-23T23:42:41 > i believe the SSL cert is specifically set up to prohibit that. if it was a > completely different domain, no problem. as a subdomain, no. > please find a relative url config (proxy pass) i can just cut/paste in afaict the certificate is not set up to prohibit that, the tls cert has no fields which could prohibit that, you're serving no http headers which could prohibit that, and you have no dns records which could prohibit that
I have another idea for setting up the proxy for the build server: have the proxy be on https://libre-soc.org:8443/ this successfully achieves having a different origin (which is why I wanted build.libre-soc.org). that way it can just use the existing tls certificate for libre-soc.org. since letsencrypt uses port 80 rather than 8443, that won't be an issue. nginx config I used for testing (I used port 10443 because my server already was using 8443 for something else): server { listen 10443 ssl; listen [::]:10443 ssl; ssl_certificate /etc/letsencrypt/live/programmerjake.tk/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/programmerjake.tk/privkey.pem; location / { proxy_pass http://jacob-build-server.programmerjake.tk/; proxy_set_header Host $host; } } modifying it for libre-soc.org: /etc/nginx/sites-available/build server { listen 8443 ssl; listen [::]:8443 ssl; ssl_certificate /path/to/libre-soc.org.crt; ssl_certificate_key /path/to/libre-soc.org.key; location / { proxy_pass http://10.6.0.7/; proxy_set_header Host $host; } }
(In reply to Jacob Lifshay from comment #53) > did more research into passing the tty devices into docker so scripts can > see the fpga's uart: > > tty devices are dynamically created and removed during the fpga programming > process -- docker's method of dealing with dynamic devices: > https://github.com/docker/cli/blob/a32cd16160f1b41c1c4ae7bee4dac929d1484e59/ > man/src/container/create-example.md#dealing-with-dynamically-created-devices- > --device-cgroup-rule > > gitlab-runner merge request for adding the device cgroup rules option: > https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309 > > afaict gitlab-runner uses docker through the go library rather than through > the docker executable, so making a wrapper script probably won't work https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3309 was merged, it should be in the next stable release at the end of this month
(In reply to Jacob Lifshay from comment #45) > (In reply to Luke Kenneth Casson Leighton from comment #41) > > urr i had to actually set up reverse-dns in bind *shudder* > > well, reverse-dns broke again: reverse-dns is still broken. maybe i should just see if i can get postfix to send mail through a ssh tunnel so libre-soc.org sees the connection coming from localhost
Since the openvpn connection between my build server and libre-soc.org is borked, I thought I'd come up with a better alternative: I created a script for setting up wireguard: https://git.libre-soc.org/?p=utils.git;a=blob;f=wg-setup.sh;h=299f5c08fc2a15112b799d69f67da132129ee322;hb=431799b73339f28ec914cea711c1af9d87533bc2 I tested it connecting my desktop and file server, trying both ways of which one is server and which one is client. The script supports multiple clients, but I didn't test that other than seeing if it does the correct modifications to the config file. I set it to use pre-shared keys as an extra layer of security on top of the public/private keys. I also set it to send keep-alive packets every 25s so it works better with NATs. Show Usage: ./wg-setup.sh It works: on both server and client: sudo apt install wireguard on the server: # create the config for wg0 that listens on # udp port 8995 (picked randomly) and has the server be 10.255.255.1 sudo ./wg-setup.sh init-server wg0 8995 10.255.255.1/32 # allow udp port 8995 through the server's firewall, # here I'm using ufw, adjust as needed. sudo ufw allow 8995/udp # create the config for the client in client.conf, adds the peering # info to the server's config for wg0. the client will initially # connect to the server at server.example.com. the client # will be 10.255.255.2. sudo ./wg-setup.sh add-client wg0 server.example.com client.conf 10.255.255.2/32 now, move client.conf to the client, chmod 600, chown root:root, and move it to /etc/wireguard/wg0.conf now, if you like systemd, run on both server and client: systemctl start wg-quick@wg0 otherwise, you can start it manually: wg-quick up wg0 or look up how wg-quick works with sysv init. If you're using openrc (idk if that's part of sysv init), afaict you can use this init script: https://gitweb.gentoo.org/repo/gentoo.git/tree/net-vpn/wireguard-tools/files/wg-quick.init and symlink to it: ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.vpn0
(In reply to Jacob Lifshay from comment #67) > If you're using openrc (idk if that's part of sysv init), afaict you can use > this init script: > https://gitweb.gentoo.org/repo/gentoo.git/tree/net-vpn/wireguard-tools/files/ > wg-quick.init > and symlink to it: > ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.vpn0 actually, that should be .wg0: ln -s /path/to/wg-quick.init /etc/init.d/wg-quick.wg0
(In reply to Jacob Lifshay from comment #67) > otherwise, you can start it manually: > wg-quick up wg0 > > or look up how wg-quick works with sysv init. I found a script that should work for sysv init: https://www.procustodibus.com/blog/2021/06/wireguard-sysv-init-script/#with-sysv-init
I decided it'd probably be better for me to just run my own server that would be at build.libre-soc.org rather than using the libre-soc.org server, that way luke and alain wouldn't be blocking progress as much, they're busy enough with their own problems already. after privately talking to luke a bit, he said that would help.
Luke, I'm getting the new server setup, can you add it to dns at build.libre-soc.org? 167.86.87.162 2a02:c207:2090:6249::1
(In reply to Jacob Lifshay from comment #71) > Luke, I'm getting the new server setup, can you add it to dns at > build.libre-soc.org? > > 167.86.87.162 > 2a02:c207:2090:6249::1 I have http[s] all set up except for getting valid certificates, that's blocked on you setting up dns.
(In reply to Jacob Lifshay from comment #72) > (In reply to Jacob Lifshay from comment #71) > > Luke, I'm getting the new server setup, can you add it to dns at > > build.libre-soc.org? > > > > 167.86.87.162 > > 2a02:c207:2090:6249::1 > > I have http[s] all set up except for getting valid certificates, that's > blocked on you setting up dns. In order to make progress, I started setting it up to use build.libre-soc.programmerjake.tk instead... I got git and https up and working. email is working but I need to get dmarc and dkim and spf working.
(In reply to Jacob Lifshay from comment #71) > Luke, I'm getting the new server setup, can you add it to dns at > build.libre-soc.org? > > 167.86.87.162 > 2a02:c207:2090:6249::1 done, host gives this: $ host build.libre-soc.org build.libre-soc.org has address 167.86.87.162 build.libre-soc.org has IPv6 address 2a02:c207:2090:6249::1
(In reply to Luke Kenneth Casson Leighton from comment #74) > (In reply to Jacob Lifshay from comment #71) > > Luke, I'm getting the new server setup, can you add it to dns at > > build.libre-soc.org? > > > > 167.86.87.162 > > 2a02:c207:2090:6249::1 > > done, host gives this: > > $ host build.libre-soc.org > build.libre-soc.org has address 167.86.87.162 > build.libre-soc.org has IPv6 address 2a02:c207:2090:6249::1 ok. I got all the https certificates set up for build.libre-soc.org and got all the email and other stuff set up for build.libre-soc.programmerjake.tk set up. I'm thinking it might be better to send all the emails and stuff from build.libre-soc.programmerjake.tk since that doesn't require your maintaining all the dns records and stuff. I enabled sending build emails for lkcl@lkcl.net and for me. I want to add a unsubscribe link, as well as making a better way to subscribe to build emails than getting one of us to ssh into the server and edit /home/build/gitlab-ci-archiver/allowed-emails.txt Anyone else who wants to get build emails, let me/luke know... the system is WIP though, so expect oddities/issues. Currently it just sends emails for build failures only to the tested commit's author and committer...any intervening commits don't get emails.
(In reply to Jacob Lifshay from comment #75) > I enabled sending build emails for lkcl@lkcl.net and for me. I want to add a > unsubscribe link, as well as making a better way to subscribe to build > emails than getting one of us to ssh into the server and edit > /home/build/gitlab-ci-archiver/allowed-emails.txt I added code for unsubscribe/resubscribe, but didn't enable it yet...review of the CGI script appreciated! https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/unsubscribe-cgi.sh Library used by it: https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/allowed-emails.sh
(In reply to Jacob Lifshay from comment #76) > I added code for unsubscribe/resubscribe, but didn't enable it yet...review > of the CGI script appreciated! After experiencing bash's propensity for inadvertently ignoring errors, which makes it particularly terrible for CGI scripts, I rewrote it in rust: https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/tree/master/email_manage that link is to both the unsubscribe cgi binary and the email-manage binary (used for changing who gets emails for which projects from the command line, as well as generating urls to access the subscription editor). they're both actually symlinks to the same file like busybox. I added per-project email subscriptions, so, for example, lkcl can be subscribed to nmutil but not soc since nmutil should pass CI so therefore he'd probably want to know when he broke it, but soc currently doesn't pass CI so the build failure messages are just annoying. I added a second email verification step to avoid the issue where build log emails get posted to mailing lists and someone clicking on your unsubscribe link to try and fiddle with all the settings, now all they'll be able to do is send you unwanted email verification emails. the verification emails are rate limited with exponential backoff, the default settings are you can send 2 emails immediately, then you have to wait 6hr, then 12hr, then 1 day, and so on till the link expires in 3 days. I figured that's a decent balance between not getting 7000 spam emails and actually being able to send yourself another verification email when necessary. I'll enable it on the build server in the morning when I can be awake in case anything breaks.
(In reply to Jacob Lifshay from comment #77) > I'll enable it on the build server in the morning when I can be awake in > case anything breaks. I enabled it and added documentation of how it works to the README.md: https://salsa.debian.org/Kazan-team/gitlab-ci-archiver/-/blob/master/README.md As far as I can tell, it works!
I'm considering this completed. If we want to add a nicer web-based git viewer, that can be done as part of a different task.
lkcl, do you think the budget for this should be increased? I did put in quite a lot of work: Code written for gitlab ci: All of: https://salsa.debian.org/Kazan-team/gitlab-ci-archiver https://git.libre-soc.org/?p=utils.git;a=shortlog;h=refs/heads/gitlab-build-utils I also modified git-mirror to use for our purposes: https://salsa.debian.org/Kazan-team/git-mirror