Bug 1162 - evaluate the cost and time of whether a git sync mirror should have time spent on it.
Summary: evaluate the cost and time of whether a git sync mirror should have time spen...
Status: IN_PROGRESS
Alias: None
Product: Libre-SOC Website
Classification: Unclassified
Component: website (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- enhancement
Assignee: Jacob Lifshay
URL:
Depends on:
Blocks:
 
Reported: 2023-09-18 20:03 BST by Jacob Lifshay
Modified: 2024-01-08 23:08 GMT (History)
3 users (show)

See Also:
NLnet milestone: ---
total budget (EUR) for completion of task and all subtasks: 0
budget (EUR) for this task, excluding subtasks' budget: 0
parent task for budget allocation:
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jacob Lifshay 2023-09-18 20:03:37 BST
https://lists.libre-soc.org/pipermail/libre-soc-dev/2023-September/005649.html

<lkcl at lkcl.net> wrote:
>
> please reduce that to 24 hours. i was not in any way consulted
> on this decision and the server being mirrored is in no position
> to sustain massive loading, plus i have to pay (personally)
> for the bandwidth.

ok, I used wireshark to measure the amount of bandwidth taken by one
sync when there are no new commits (which is just a series of no-op
git fetches), it comes out to 669838 bytes. calculating based on the
hosting provider's published rate of 5 GBP / TB bandwidth (which is
overcounting since afaict they count the max of upload and download
bandwidth rather than the sum):
(669838*byte) * 1/(15*min) * (60*min/hr) * (24*hr/day) *
(365.2425*day/yr) * (5*gbp)/(1e12*byte) gives:
0.117*gbp/yr (rounded to 3 decimals)

this is a trivial enough amount that I can pay you 10x that per year
if you like...

the total amount of time used per sync was 55s

can I set it to at least once per hour since this is the exact same
git sync that runs our CI and people depend on CI being somewhat
timely for their work, I know at least ghostmansd and I often look at
the CI right after we push stuff.
Comment 1 Jacob Lifshay 2023-09-18 21:01:34 BST
I have turned off the git mirror for now...
Comment 2 Luke Kenneth Casson Leighton 2023-09-18 21:24:46 BST
i was deeply unimpressed that there is the stupid belief that i
have set up libresoc on a stupid raspberry pi.  this would be incredibly
irresponsible and place the entire project at risk.

if anyone is actually prepared to ask what the actual situation is
then i will describe it and you can spend the time documenting it.A

THEN the cost and effort of whether it is worthwhile creating mirrors can
be evaluated.
Comment 3 Luke Kenneth Casson Leighton 2023-09-18 21:25:23 BST
(In reply to Jacob Lifshay from comment #0)

> can I set it to at least once per hour 

no you may not.
Comment 4 Jacob Lifshay 2023-09-18 21:42:27 BST
(In reply to Luke Kenneth Casson Leighton from comment #2)
> i was deeply unimpressed that there is the stupid belief that i
> have set up libresoc on a stupid raspberry pi.  this would be incredibly
> irresponsible and place the entire project at risk.

I did not assume that and I am not aware of anyone expressing that belief.
> 
> if anyone is actually prepared to ask what the actual situation is
> then i will describe it and you can spend the time documenting it.

from what I understand, you have it hosted on a mythic beasts VPS, with manual backups to your laptop. feel free to elaborate, if you like.

> 
> THEN the cost and effort of whether it is worthwhile creating mirrors can
> be evaluated.

I think having git mirrors is useful for 4 reasons:
1. it is what drives CI, eventually having CI on more repos is imho a good idea (the additional repos are just mirrors for now).

2. salsa.debian.org has a much nicer interface, so things like git blame actually can be used there rather than having to download the git repo and run blame locally (even then, local git blame has a worse UI imo). There are also other nice features, such as being able to easily compare two branches/tags/commits without having to manually type in the proper url as is needed for git.libre-soc.org.

3. it increases libre-soc.org's visibility somewhat.

4. if libre-soc.org goes down or is blocked, salsa.debian.org likely won't simultaneously be down/blocked
Comment 5 Jacob Lifshay 2023-09-19 01:16:36 BST
I will note that the amount of work I have to do to change the period is trivial...just change 2 lines of code, commit it, and install the new commit to my server.
Comment 6 Jacob Lifshay 2023-09-19 22:20:47 BST
Luke, I'm sorry for setting up the additional mirrors without consulting you. we'll talk about this more later
Comment 7 Jacob Lifshay 2023-11-06 04:50:50 GMT
I enabled the git-mirror service so CI works again. I set it to run once every 24hr as luke stipulated (though that is massively inconvenient), I also changed it to sleep 5s between each git fetch:

https://salsa.debian.org/Kazan-team/git-mirror/-/commits/9573af655a7797a138b5bbd500fdd0b28332b51a

commit 9573af655a7797a138b5bbd500fdd0b28332b51a
Author: Jacob Lifshay <programmerjake@gmail.com>
Date:   Sun Nov 5 20:36:13 2023 -0800

    change update period to 24hr

commit 1c1371eebe7f440b5f323985e9bd0e6984f9bef9
Author: Jacob Lifshay <programmerjake@gmail.com>
Date:   Sun Nov 5 20:35:38 2023 -0800

    sleep 5sec between updating each git repo