Bug 181 - test and install public-inbox
Summary: test and install public-inbox
Status: RESOLVED FIXED
Alias: None
Product: Libre-SOC Website
Classification: Unclassified
Component: website (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- enhancement
Assignee: vklr@vkten.in
URL:
Depends on:
Blocks:
 
Reported: 2020-02-15 13:53 GMT by Luke Kenneth Casson Leighton
Modified: 2020-03-21 07:01 GMT (History)
4 users (show)

See Also:
NLnet milestone: ---
total budget (EUR) for completion of task and all subtasks: 0
budget (EUR) for this task, excluding subtasks' budget: 0
parent task for budget allocation:
child tasks for budget allocation:
The table of payments (in EUR) for this task; TOML format:


Attachments
Public-inbox (180 bytes, text/plain)
2020-03-17 01:24 GMT, vklr@vkten.in
Details
Public-inbox 1 (180 bytes, text/plain)
2020-03-17 01:28 GMT, vklr@vkten.in
Details
Public-inbox 2 (657 bytes, text/plain)
2020-03-17 01:29 GMT, vklr@vkten.in
Details
Public-inbox 3 (865 bytes, text/plain)
2020-03-17 01:30 GMT, vklr@vkten.in
Details
Public-inbox 4 (133 bytes, text/plain)
2020-03-17 01:31 GMT, vklr@vkten.in
Details
Public-inbox 5 (209 bytes, text/plain)
2020-03-17 01:31 GMT, vklr@vkten.in
Details
New detailed setup instructions (5.28 KB, text/plain)
2020-03-20 12:48 GMT, vklr@vkten.in
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Luke Kenneth Casson Leighton 2020-02-15 13:53:39 GMT
needs to be installed, however exim4 + mailman2 needs to be converted
to Maildir, first.

assume debian/10 base OS.
Comment 1 Luke Kenneth Casson Leighton 2020-02-15 14:05:29 GMT
so if you can set up a "default" exim4 system and just assume that
the sole exclusive purpose for which it is to be used is for
mailman2

(note: *not* mailman3)

then, once that is working, convert it to use Maildir, and let me know
how that's done.

the instructions need to be *real* simple.
Comment 2 vklr@vkten.in 2020-02-15 17:53:46 GMT
I will start work on this.
Comment 3 vklr@vkten.in 2020-02-17 11:30:22 GMT
The relevant flows for exim4 + mailman2 + public-inbox setup

1) exim4 -> Maildir
   mailman2 -> exim4
   public-inbox
     public-inbox-mda -> needs msg from stdin
       mailman2 external archiver method can be used
        or
       may be some way of getting input from exim4
      or
     public-inbox-watch -> needs the exim4 Maildir

2) exim4 -> Maildir
   mailman2 -> Reads from exim4 Maildir (Experimental)
   public-inbox
    same as above
Comment 4 vklr@vkten.in 2020-02-17 11:35:18 GMT
So Luke, should I have to only show you real simple instructions for setting up exim4 with Maildir as the mail output format?

mailman2 supports reading from Maildir but it is experimental.
mailman3 removed even this experimental support, just for info.
Comment 5 Luke Kenneth Casson Leighton 2020-02-17 11:56:39 GMT
(In reply to vklr@vkten.in from comment #4)
> So Luke, should I have to only show you real simple instructions for setting
> up exim4 with Maildir as the mail output format?

yes please.  and how to configure Mailman2 to *accept* Maildir format

if i convert exim4 to drop into Maildir *without* also telling Mailman2 to read it, the lists entirely break.

this is a live list so i cannot take risks.

changing two things at once makes me extremely nervous.

therefore i need a *tested* setup.


> mailman2 supports reading from Maildir but it is experimental.
> mailman3 removed even this experimental support, just for info.

argh.

have to raise that as a bug.
Comment 6 Luke Kenneth Casson Leighton 2020-02-17 12:00:10 GMT
(In reply to vklr@vkten.in from comment #3)
> The relevant flows for exim4 + mailman2 + public-inbox setup
> 
> 1) exim4 -> Maildir
>    mailman2 -> exim4

it is more exim4 -> mailman2.

>    public-inbox
>      public-inbox-mda -> needs msg from stdin
>        mailman2 external archiver method can be used

ah what is that? can you show links etc about that?

>         or
>        may be some way of getting input from exim4
>       or
>      public-inbox-watch -> needs the exim4 Maildir

unfortunately we need to extract the mbox formatted archives as well. i have seen mbox 2 maildir converters, online.

> 2) exim4 -> Maildir
>    mailman2 -> Reads from exim4 Maildir (Experimental)
>    public-inbox
>     same as above
Comment 7 vklr@vkten.in 2020-02-17 12:47:07 GMT
One thing I forgot to mention that reading Maildir using mailman2 requires running as root.

(In reply to Luke Kenneth Casson Leighton from comment #6)
> (In reply to vklr@vkten.in from comment #3)
> > The relevant flows for exim4 + mailman2 + public-inbox setup
> > 
> > 1) exim4 -> Maildir
> >    mailman2 -> exim4
> 
> it is more exim4 -> mailman2.

Yeah, just used as some sort of connection indicator.
Should have used "---" instead.

> 
> >    public-inbox
> >      public-inbox-mda -> needs msg from stdin
> >        mailman2 external archiver method can be used
> 
> ah what is that? can you show links etc about that?
> 

About public-inbox:
<link https://public-inbox.org/>
<link https://public-inbox.org/flow.html>
<link https://public-inbox.org/public-inbox-mda.html>
<link https://public-inbox.org/public-inbox-watch.html>

Are you also asking info about mailman2 external archiver method?

> >         or
> >        may be some way of getting input from exim4
> >       or
> >      public-inbox-watch -> needs the exim4 Maildir
> 
> unfortunately we need to extract the mbox formatted archives as well. i have
> seen mbox 2 maildir converters, online.
> 

mb2md does this.
<link https://packages.debian.org/buster/mb2md>
<link http://batleth.sapienti-sat.org/projects/mb2md/> Homepage

Also can you give me the low level flow detail between exim4 and mailman2?
I mean how they exchange data. As I can't make out from installation
instructions. They are abstract. Quick read did not showed it.
Or else I will read, install and read more about it.
Comment 8 vklr@vkten.in 2020-02-17 12:54:15 GMT
(In reply to Luke Kenneth Casson Leighton from comment #5)
> (In reply to vklr@vkten.in from comment #4)
> 
> > mailman2 supports reading from Maildir but it is experimental.
> > mailman3 removed even this experimental support, just for info.
> 
> argh.
> 
> have to raise that as a bug.

In mailman2, the Maildir support was made somewhere in 2014.
In mailman3, Maildir support was removed and instead LMTP was used
between exim4 and mailman2.
Comment 9 Luke Kenneth Casson Leighton 2020-02-17 13:05:43 GMT
(In reply to vklr@vkten.in from comment #7)
> One thing I forgot to mention that reading Maildir using mailman2 requires
> running as root.
> 
> (In reply to Luke Kenneth Casson Leighton from comment #6)
> > (In reply to vklr@vkten.in from comment #3)
> > > The relevant flows for exim4 + mailman2 + public-inbox setup
> > > 
> > > 1) exim4 -> Maildir
> > >    mailman2 -> exim4
> > 
> > it is more exim4 -> mailman2.
> 
> Yeah, just used as some sort of connection indicator.
> Should have used "---" instead.
> 
> > 
> > >    public-inbox
> > >      public-inbox-mda -> needs msg from stdin
> > >        mailman2 external archiver method can be used
> > 
> > ah what is that? can you show links etc about that?
> > 
> 
> About public-inbox:
> <link https://public-inbox.org/>
> <link https://public-inbox.org/flow.html>
> <link https://public-inbox.org/public-inbox-mda.html>
> <link https://public-inbox.org/public-inbox-watch.html>
> 
> Are you also asking info about mailman2 external archiver method?

yes.  if that "works" - by public-inbox literally parsing the pipermail
list structure, that's fantastic.

however...

i *believe*... after talking to Eric... that even that method expects to
see the archives (the actual messages) in "Maildir" format,
*even over the internet i.e. via HTTP*.

whereas what pipermail does - i think - is expose the mbox contents, directly, 
on a monthly basis.

i could be wrong: i don't honestly know.

> 
> > >         or
> > >        may be some way of getting input from exim4
> > >       or
> > >      public-inbox-watch -> needs the exim4 Maildir
> > 
> > unfortunately we need to extract the mbox formatted archives as well. i have
> > seen mbox 2 maildir converters, online.
> > 
> 
> mb2md does this.
> <link https://packages.debian.org/buster/mb2md>
> <link http://batleth.sapienti-sat.org/projects/mb2md/> Homepage
> 
> Also can you give me the low level flow detail between exim4 and mailman2?

i can't - i literally did "apt-get install mailman2" and followed
online instructions on "debian exim4 mailman2 howto" and that was it.

i didn't *understand* those instructions, i simply followed them.

> I mean how they exchange data. 

i *believe* - infer / deduce / hypothesise - that "mbox" format is used.
namely, that exim4 gets configured to "drop the incoming list email
into an mbox-formatted file" which mailman2 is then (separately,
in mm_config.py) configured (as per instructions which i read and followed
to the letter, mentioned above) to monitor.

> As I can't make out from installation
> instructions. They are abstract. Quick read did not showed it.
> Or else I will read, install and read more about it.

now you get why i am reluctant to mess with the live setup.
Comment 10 Luke Kenneth Casson Leighton 2020-02-17 13:07:02 GMT
(In reply to vklr@vkten.in from comment #8)

> In mailman2, the Maildir support was made somewhere in 2014.
> In mailman3, Maildir support was removed and instead LMTP was used
> between exim4 and mailman2.

that makes sense from a technical perspective however it is then a bitch
to integrate public-inbox.

the point of using the Maildir format is to point public-inbox *directly*
at the output from exim4.
Comment 11 vklr@vkten.in 2020-02-17 13:35:22 GMT
(In reply to Luke Kenneth Casson Leighton from comment #9)
> (In reply to vklr@vkten.in from comment #7)
> > One thing I forgot to mention that reading Maildir using mailman2 requires
> > running as root.
> > 
> > (In reply to Luke Kenneth Casson Leighton from comment #6)
> > > (In reply to vklr@vkten.in from comment #3)
> > > > The relevant flows for exim4 + mailman2 + public-inbox setup
> > 
> > > 
> > > >    public-inbox
> > > >      public-inbox-mda -> needs msg from stdin
> > > >        mailman2 external archiver method can be used
> > > 
> > > ah what is that? can you show links etc about that?
> > > 
> > 
> > Are you also asking info about mailman2 external archiver method?
> 
> yes.  if that "works" - by public-inbox literally parsing the pipermail
> list structure, that's fantastic.
> 
> however...
> 
> i *believe*... after talking to Eric... that even that method expects to
> see the archives (the actual messages) in "Maildir" format,
> *even over the internet i.e. via HTTP*.
> 

Who is Eric?
Comment 12 vklr@vkten.in 2020-02-17 13:37:30 GMT
(In reply to Luke Kenneth Casson Leighton from comment #10)
> (In reply to vklr@vkten.in from comment #8)
> 
> > that makes sense from a technical perspective however it is then a bitch
> to integrate public-inbox.
> 
> the point of using the Maildir format is to point public-inbox *directly*
> at the output from exim4.

Do you want pipermail function as well?
Comment 13 Luke Kenneth Casson Leighton 2020-02-17 13:54:06 GMT
(In reply to vklr@vkten.in from comment #12)

> Do you want pipermail function as well?

yes, pipermail functionality should not be touched because it's what people
expect (and the google search archives have already picked it up)
Comment 14 Luke Kenneth Casson Leighton 2020-02-17 16:09:32 GMT
(In reply to vklr@vkten.in from comment #11)

> > i *believe*... after talking to Eric... that even that method expects to
> > see the archives (the actual messages) in "Maildir" format,
> > *even over the internet i.e. via HTTP*.
> > 
> 
> Who is Eric?


sorry, eric is the maintainer of public-inbox.
Comment 15 vklr@vkten.in 2020-02-19 02:44:57 GMT
(In reply to Luke Kenneth Casson Leighton from comment #10)
> (In reply to vklr@vkten.in from comment #8)
> 
> the point of using the Maildir format is to point public-inbox *directly*
> at the output from exim4.

Do you want public-inbox to get input directly from exim4 output or from mailman2 output (that is after mailman2 processing; e.g. spam, unseen addresses).
Comment 16 Luke Kenneth Casson Leighton 2020-02-19 05:19:39 GMT
(In reply to vklr@vkten.in from comment #15)
> (In reply to Luke Kenneth Casson Leighton from comment #10)
> > (In reply to vklr@vkten.in from comment #8)
> > 
> > the point of using the Maildir format is to point public-inbox *directly*
> > at the output from exim4.
> 
> Do you want public-inbox to get input directly from exim4 output or from
> mailman2 output (that is after mailman2 processing; e.g. spam, unseen
> addresses).

honestly i just want sonething that works! :)
Comment 17 Yehowshua 2020-03-01 18:17:35 GMT
Just out of curiosity, what exactly is a public inbox?
Comment 18 Luke Kenneth Casson Leighton 2020-03-01 19:16:58 GMT
(In reply to Yehowshua from comment #17)
> Just out of curiosity, what exactly is a public inbox?

https://try.public-inbox.org/README
Comment 19 Yehowshua 2020-03-01 19:19:21 GMT
I guess what I meant to ask is what does this do that archives currently don't?
Looks like you can add a searchable function...

Would we migrate the current archives to the public inbox?
Comment 20 Luke Kenneth Casson Leighton 2020-03-01 19:23:20 GMT
(In reply to Yehowshua from comment #19)
> I guess what I meant to ask is what does this do that archives currently
> don't?

they're backed by a git repository which means that the entire archives
are offline...

> Looks like you can add a searchable function...

... and therefore offline searchable by installing public-inbox locally
and using "git pull" on the inbox-repo.

right now, for example, that would be incredibly useful.  i'm using an
extremely unreliable 3G/4G-to-WIFI connection because the nearest cell
tower (quantity 1) is over the brow of a hill covered in trees, where
the weather (concentration of water molecules) adversely and directly
affects the signal.


> Would we migrate the current archives to the public inbox?

well, i'm kinda expecting that public-inbox simply regularly slurps
from mailman2's pipermail archives, so that it's just a matter of
"continuous constant migration"
Comment 21 vklr@vkten.in 2020-03-06 00:08:34 GMT
What use cases are you thinking of public-inbox?

Luke said offline git search.

public-inbox provides:
1) http web access
2) Atom feed
3) NNTP feed

Specify please?
Comment 22 Luke Kenneth Casson Leighton 2020-03-06 00:50:48 GMT
(In reply to vklr@vkten.in from comment #21)
> What use cases are you thinking of public-inbox?
> 
> Luke said offline git search.
> 
> public-inbox provides:
> 1) http web access
> 2) Atom feed
> 3) NNTP feed
> 
> Specify please?

really, only the http web access would be needed (i would install it locally on my laptop and access it locally)

we _could_ i imagine set up public-inbox actually on libre-riscv.org itself.

mainly i want to be able to publish the git repository of the mailing list, via git.libre-riscv.org
Comment 23 Luke Kenneth Casson Leighton 2020-03-11 12:00:21 GMT
okaaaay i think i finally got a clue:
https://public-inbox.org/meta/CAPweEDyK7FZKCk3rszy2-7ODQ5=NWAy4aaG84Rx6pc1qoixzTQ@mail.gmail.com/T/#u

i'm currently attempting to set this up as it does "no damage",
where previously i thought that it required significant modifications
to exim4 and mailman.
Comment 24 Luke Kenneth Casson Leighton 2020-03-11 12:02:56 GMT
test please ignore
Comment 25 Luke Kenneth Casson Leighton 2020-03-11 12:34:35 GMT
https://git.libre-riscv.org/?p=mailman.git;a=tree

success!  a test list, which i'm going to use an excuse to do another
test...
Comment 27 Luke Kenneth Casson Leighton 2020-03-11 14:37:01 GMT
(In reply to Jacob Lifshay from comment #26)
> note that the list password for publicinbox@libre-riscv.org is exposed:

ah well done me! :)  well, the plan is to move to libre-soc with new mailing
lists anyway.
Comment 28 Jacob Lifshay 2020-03-11 14:40:10 GMT
(In reply to Luke Kenneth Casson Leighton from comment #27)
> (In reply to Jacob Lifshay from comment #26)
> > note that the list password for publicinbox@libre-riscv.org is exposed:
> 
> ah well done me! :)  well, the plan is to move to libre-soc with new mailing
> lists anyway.

I think this is because the git repo contains everything sent by mailman as well as everything received, where it shouldn't include email generated by mailman.
Comment 29 Luke Kenneth Casson Leighton 2020-03-11 14:55:50 GMT
(In reply to Jacob Lifshay from comment #28)

> I think this is because the git repo contains everything sent by mailman as
> well as everything received, where it shouldn't include email generated by
> mailman.

it's configured as a "subscribed user", just like all other mailman users, and that "user" happens to be an account that happens to be local on the host that happens to be named "libre-riscv.org".  therefore, *all* mail sent by mailman goes *to* that user - publicinbox@libre-riscv.org - it ends up in /home/publicinbox/Maildir - and public-inbox-watch picks it up from there.

however only things that have the list-id tag are supposed to show up: i configured public-inbox to "filter" messages only having that tag (otherwise messages from other lists end up in the git repo).

it looks like the tag may have been included even when mailman sent "admin" and "subscribe" messages.
Comment 30 vklr@vkten.in 2020-03-12 01:31:40 GMT
Well. The way I thought was to subscribe to mailing list and have that delivered
to a local user, but filter that via exim4 redirection to public-inbox-mda.

No need for offlineimap, public-inbox-mirror needed only for initialization only.
No need for another copy of mails in Maildir directory and no need of cron job.

I am in testing phase with above setup.
Comment 31 Luke Kenneth Casson Leighton 2020-03-12 01:58:42 GMT
(In reply to vklr@vkten.in from comment #30)
> Well. The way I thought was to subscribe to mailing list and have that
> delivered
> to a local user, but filter that via exim4 redirection to public-inbox-mda.

ah i would be much happier with something that did not need Maildir or the cron job (or took up extra disk space)

the local user is up and running (as you can see)

> No need for offlineimap, public-inbox-mirror needed only for initialization
> only.
> No need for another copy of mails in Maildir directory and no need of cron
> job.
> 
> I am in testing phase with above setup.

fantastic, sounds much better, and i think eric will really like it.
Comment 32 vklr@vkten.in 2020-03-17 01:24:46 GMT
Created attachment 38 [details]
Public-inbox

pi.txt
Comment 33 vklr@vkten.in 2020-03-17 01:25:29 GMT
I have worked out a basic exim4-mailman2-public-inbox setup.

Please see the attachments for details.
pi.txt, pi-init.txt, exim4.txt, git.txt and pi-mda.txt.

Please note only mailing list messages are delivered.
Read pi-mda.txt

Exim4 can be set to deliver copy of mails to the subscribed email account to separate Maildir directory or Mbox file.

And as Eric said a cron job can delete old files in case of Maildir.
Dovecot can be setup to deliver mails from this folder.

As for conversion of old Mbox files to public-inbox, am working on it.
Debian mb2md package can do it. public-inbox-watch can be used for
initialization from this Maildir data. And then regular public-inbox-mda does
its job of live delivery.
Comment 34 vklr@vkten.in 2020-03-17 01:28:01 GMT
Created attachment 39 [details]
Public-inbox 1

pi.txt
Comment 35 vklr@vkten.in 2020-03-17 01:29:28 GMT
Created attachment 40 [details]
Public-inbox 2

pi-init.txt
Comment 36 vklr@vkten.in 2020-03-17 01:30:27 GMT
Created attachment 41 [details]
Public-inbox 3

exim4.txt
Comment 37 vklr@vkten.in 2020-03-17 01:31:05 GMT
Created attachment 42 [details]
Public-inbox 4

git.txt
Comment 38 vklr@vkten.in 2020-03-17 01:31:45 GMT
Created attachment 43 [details]
Public-inbox 5

pi-mda.txt
Comment 39 Luke Kenneth Casson Leighton 2020-03-17 07:49:53 GMT
(In reply to vklr@vkten.in from comment #33)
> I have worked out a basic exim4-mailman2-public-inbox setup.
> 
> Please see the attachments for details.
> pi.txt, pi-init.txt, exim4.txt, git.txt and pi-mda.txt.
> 
> Please note only mailing list messages are delivered.
> Read pi-mda.txt

will do, thank you veera.

> Exim4 can be set to deliver copy of mails to the subscribed email account to
> separate Maildir directory or Mbox file.

i just simply converted the entirety of exim4, globally, to Maildir, because i do not permit (or use) exim4 for local mail.  this is the only exception.


> And as Eric said a cron job can delete old files in case of Maildir.
> Dovecot can be setup to deliver mails from this folder.

the server does not have *any* local mailboxes except the public-inbox one (in the current experimental setup).

it is the first time, ever, since 2008 that i have allowed exim4 to have "local mail delivery".

 
> As for conversion of old Mbox files to public-inbox, am working on it.
> Debian mb2md package can do it. public-inbox-watch can be used for
> initialization from this Maildir data. And then regular public-inbox-mda does
> its job of live delivery.

fantastic.

ok so we should probably retire libre-riscv-dev and use new list names (on the new domain).  that makes life a lot easier.
Comment 40 Luke Kenneth Casson Leighton 2020-03-17 07:58:03 GMT
exim4: routers - please note order of routers is important
 otherwise mail would reach one and not other router
890_local_pi
900_exim4-config_local_user

veera did you modify 900_exim4-config_local_user?

also... will this make *all* mail go into the "public_inbox" section?
(890-local_pi)?

or is it, because it is "local_parts=test", that is a "filter" which
captures just that email address, test@test.local, correct?

so i will need to change that to:

    local_parts=libre-riscv-dev

and domains to:

    domains = lists.libre-riscv.org

and all other domains (of which there are a few) will be left alone, correct?

how do you specify multiple local_parts?  we will have multiple lists.
is it

    local_parts=[libre-riscv-dev, libre-riscv-misc, ....]

i am not sure of the syntax.
Comment 41 vklr@vkten.in 2020-03-17 09:22:51 GMT
(In reply to Luke Kenneth Casson Leighton from comment #40)
> exim4: routers - please note order of routers is important
>  otherwise mail would reach one and not other router
> 890_local_pi
> 900_exim4-config_local_user
> 
> veera did you modify 900_exim4-config_local_user?
> 

No!

890_local_pi takes all the mail for test@test.local before 900_exim4-config_local_user does. See test is local user, so by default it gets
delivered to mail_spool by 900_exim4... router.

> also... will this make *all* mail go into the "public_inbox" section?
> (890-local_pi)?
> 
> or is it, because it is "local_parts=test", that is a "filter" which
> captures just that email address, test@test.local, correct?
> 

Yes. Just this test@test.local email address.

local_parts is one before @
domains in one (full) after @ i.e test.local

> so i will need to change that to:
> 
>     local_parts=libre-riscv-dev
> 

Yes. Exactly.

> and domains to:
> 
>     domains = lists.libre-riscv.org
> 

Yes.

> and all other domains (of which there are a few) will be left alone, correct?
> 

Yes, correct.

> how do you specify multiple local_parts?  we will have multiple lists.
> is it
> 
>     local_parts=[libre-riscv-dev, libre-riscv-misc, ....]
> 

exim4 lists:

local_parts = libre-riscv-dev : libre-riscv-misc : abc : def

Must give a space before and after the colon.

> i am not sure of the syntax.
Comment 42 Luke Kenneth Casson Leighton 2020-03-17 10:44:57 GMT
(In reply to vklr@vkten.in from comment #41)

> > veera did you modify 900_exim4-config_local_user?
> > 
> 
> No!

ok.  just double-checking.

> > and all other domains (of which there are a few) will be left alone, correct?
> > 
> 
> Yes, correct.

excellent! i will try it out, now.
 
> > how do you specify multiple local_parts?  we will have multiple lists.

> exim4 lists:
> 
> local_parts = libre-riscv-dev : libre-riscv-misc : abc : def
> 
> Must give a space before and after the colon.

ah :)  thank you.
Comment 43 Luke Kenneth Casson Leighton 2020-03-17 10:56:36 GMT
wait... hang on... i should check: this should be checking for what
goes to the *user that is subscribed to the list*.

not the actual list itself, is that correct?

so:

public_inbox:
    debug_print = "R: public_inbox for $local_part@$domain"
    driver = accept
#   require_files = PI_HOME/check.pck
    domains = libre-riscv.org
    check_local_user
    local_parts = publicinbox
    transport = public_inbox_transport

not:

    domains = lists.libre-riscv.org
    check_local_user
    local_parts = libre-riscv-dev

this latter i believe would take *all* email sent to libre-riscv-dev@lists.libre-riscv.org and push it into public-inbox

whereas what we want is the mail delivered *from mailman*, where "publicinbox@libre-riscv.org" happens to be one of the recipients.
Comment 44 Luke Kenneth Casson Leighton 2020-03-17 11:12:17 GMT
ok i think i have it working (tested with mailman@lists.libre-riscv.org)
this is a test of the list
Comment 45 Luke Kenneth Casson Leighton 2020-03-17 13:49:01 GMT
yippee!  thank you, veera.  i made one small change to the config:
set umask=066 because after writing to /home/publicinbox/libre-riscv-dev.git
if the umask is 077 then gitweb will not be able to see the files for
publishing them via git.libre-riscv.org

wha-hey, it works!

ok just for a laugh i'm going to run public-inbox-httpd, which is
now set up on http://inbox.libre-riscv.org/libre-riscv-dev
Comment 46 vklr@vkten.in 2020-03-17 14:01:38 GMT
(In reply to Luke Kenneth Casson Leighton from comment #43)
> wait... hang on... i should check: this should be checking for what
> goes to the *user that is subscribed to the list*.
> 
> not the actual list itself, is that correct?
> 
> so:
> 
> public_inbox:
>     debug_print = "R: public_inbox for $local_part@$domain"
>     driver = accept
> #   require_files = PI_HOME/check.pck
>     domains = libre-riscv.org
>     check_local_user
>     local_parts = publicinbox
>     transport = public_inbox_transport
> 
> not:
> 
>     domains = lists.libre-riscv.org
>     check_local_user
>     local_parts = libre-riscv-dev
> 
> this latter i believe would take *all* email sent to
> libre-riscv-dev@lists.libre-riscv.org and push it into public-inbox
> 

Yes.

> whereas what we want is the mail delivered *from mailman*, where
> "publicinbox@libre-riscv.org" happens to be one of the recipients.

Yes.
Comment 47 Luke Kenneth Casson Leighton 2020-03-17 14:03:28 GMT
(In reply to vklr@vkten.in from comment #46)

> > whereas what we want is the mail delivered *from mailman*, where
> > "publicinbox@libre-riscv.org" happens to be one of the recipients.
> 
> Yes.

thanks veera.  all good!  really appreciate your help on this.  sorry it took me a while to work out what was needed.
Comment 48 vklr@vkten.in 2020-03-17 14:12:05 GMT
(In reply to Luke Kenneth Casson Leighton from comment #45)
> yippee!  thank you, veera.  i made one small change to the config:
> set umask=066 because after writing to /home/publicinbox/libre-riscv-dev.git
> if the umask is 077 then gitweb will not be able to see the files for
> publishing them via git.libre-riscv.org
> 
Yes. You have to set as your requiremnet.
> wha-hey, it works!
> 
> ok just for a laugh i'm going to run public-inbox-httpd, which is
> now set up on http://inbox.libre-riscv.org/libre-riscv-dev
So you are using subdomains by adding A records in DNS.
You can also use nginx reverse proxy so that http://libre-riscv.org:8080/pi/libre-riscv-dev becomes some path as http://libre-riscv.org:80/pi/libre-riscv-dev
Comment 49 Luke Kenneth Casson Leighton 2020-03-17 14:43:26 GMT
(In reply to vklr@vkten.in from comment #48)

> > ok just for a laugh i'm going to run public-inbox-httpd, which is
> > now set up on http://inbox.libre-riscv.org/libre-riscv-dev
> So you are using subdomains by adding A records in DNS.

yes.  i found that proxy redirect would send to the wrong
path.  i don't know why.

> You can also use nginx reverse proxy so that
> http://libre-riscv.org:8080/pi/libre-riscv-dev becomes some path as
> http://libre-riscv.org:80/pi/libre-riscv-dev

i had to use this:

        location /{
                access_log /var/log/nginx/public-inbox-httpd_access.log;
                error_log /var/log/nginx/public-inbox-httpd_error.log;
                proxy_set_header    HOST $host;
                proxy_set_header    X-Real-IP $remote_addr;
                proxy_set_header    X-Forwarded-Proto $scheme;
                proxy_pass         http://127.0.0.1:8080$request_uri;
                proxy_buffering off;
        }

efforts to change it to work as location /publicinbox met with...
resistance :)
Comment 50 vklr@vkten.in 2020-03-20 12:46:55 GMT
Public-inbox Rev 2

Hey I worked out setup from Mailman2 mbox archives to public-inbox.

Steps are in the attachment.

pi-rev2.txt
Comment 51 vklr@vkten.in 2020-03-20 12:48:52 GMT
Created attachment 46 [details]
New detailed setup instructions

pi-rev2.txt

New detailed instructions for setup.
Comment 52 vklr@vkten.in 2020-03-20 12:51:24 GMT
Addition
--------

For public-inbox-watch always use listid option
in ~/.public-inbox/config

So that it picks up only mailing list posted emails only.
Comment 53 Luke Kenneth Casson Leighton 2020-03-20 18:27:26 GMT
(In reply to vklr@vkten.in from comment #51)
> Created attachment 46 [details]
> New detailed setup instructions
> 
> pi-rev2.txt
> 
> New detailed instructions for setup.

fantastic.  modified accordingly.

for next time (libre-soc lists) we definitely want to use -V2.
Comment 54 vklr@vkten.in 2020-03-21 00:28:40 GMT
Forgot to add.

Please edit: test1.git/description

So that nice description is shown by public-inbox-httpd
Comment 55 vklr@vkten.in 2020-03-21 01:13:01 GMT
(In reply to Luke Kenneth Casson Leighton from comment #49)
> (In reply to vklr@vkten.in from comment #48)
> 
> > > ok just for a laugh i'm going to run public-inbox-httpd, which is
> > > now set up on http://inbox.libre-riscv.org/libre-riscv-dev
> > So you are using subdomains by adding A records in DNS.
> 
> yes.  i found that proxy redirect would send to the wrong
> path.  i don't know why.
> 
> > You can also use nginx reverse proxy so that
> > http://libre-riscv.org:8080/pi/libre-riscv-dev becomes some path as
> > http://libre-riscv.org:80/pi/libre-riscv-dev
> 
> i had to use this:
> 
>         location /{
>                 access_log /var/log/nginx/public-inbox-httpd_access.log;
>                 error_log /var/log/nginx/public-inbox-httpd_error.log;
>                 proxy_set_header    HOST $host;
>                 proxy_set_header    X-Real-IP $remote_addr;
>                 proxy_set_header    X-Forwarded-Proto $scheme;
>                 proxy_pass         http://127.0.0.1:8080$request_uri;
>                 proxy_buffering off;
>         }
> 
> efforts to change it to work as location /publicinbox met with...
> resistance :)

please try this:

       location /publicinbox/ {
                 access_log /var/log/nginx/public-inbox-httpd_access.log;
                 error_log /var/log/nginx/public-inbox-httpd_error.log;
                 proxy_pass         https://127.0.0.1:8080/;
                 proxy_buffering off;
         }

Also please install libplack-middleware-reverseproxy-perl
It's needed with nginx.
Comment 56 vklr@vkten.in 2020-03-21 01:14:51 GMT
> please try this:
> 
>        location /publicinbox/ {
>                  access_log /var/log/nginx/public-inbox-httpd_access.log;
>                  error_log /var/log/nginx/public-inbox-httpd_error.log;
>                  proxy_pass         https://127.0.0.1:8080/;
>                  proxy_buffering off;
>          }
> 
> Also please install libplack-middleware-reverseproxy-perl
> It's needed with nginx.

this location directive within server directive for http://libre-riscv.org/
Comment 57 Luke Kenneth Casson Leighton 2020-03-21 04:18:07 GMT
(In reply to vklr@vkten.in from comment #55)
> (In reply to Luke Kenneth Casson Leighton from comment #49)
> > (In reply to vklr@vkten.in from comment #48)
> > 
> > > > ok just for a laugh i'm going to run public-inbox-httpd, which is
> > > > now set up on http://inbox.libre-riscv.org/libre-riscv-dev
> > > So you are using subdomains by adding A records in DNS.
> > 
> > yes.  i found that proxy redirect would send to the wrong
> > path.  i don't know why.
> > 
> > > You can also use nginx reverse proxy so that
> > > http://libre-riscv.org:8080/pi/libre-riscv-dev becomes some path as
> > > http://libre-riscv.org:80/pi/libre-riscv-dev
> > 
> > i had to use this:
> > 
> >         location /{
> >                 access_log /var/log/nginx/public-inbox-httpd_access.log;
> >                 error_log /var/log/nginx/public-inbox-httpd_error.log;
> >                 proxy_set_header    HOST $host;
> >                 proxy_set_header    X-Real-IP $remote_addr;
> >                 proxy_set_header    X-Forwarded-Proto $scheme;
> >                 proxy_pass         http://127.0.0.1:8080$request_uri;
> >                 proxy_buffering off;
> >         }
> > 
> > efforts to change it to work as location /publicinbox met with...
> > resistance :)
> 
> please try this:
> 
>        location /publicinbox/ {
>                  access_log /var/log/nginx/public-inbox-httpd_access.log;
>                  error_log /var/log/nginx/public-inbox-httpd_error.log;
>                  proxy_pass         https://127.0.0.1:8080/;
>                  proxy_buffering off;
>          }

this fails not in nginx but in public-inbox-httpd.  the URL is passed in full including the word "publicinbox" and it cannot cope, it starts looking for a list named "publicinbox".

a rewrite rule or double proxypass is needed.

this is why i went with the domain name

> Also please install libplack-middleware-reverseproxy-perl
> It's needed with nginx.

i have that.
Comment 58 vklr@vkten.in 2020-03-21 07:01:25 GMT
(In reply to Luke Kenneth Casson Leighton from comment #57)
> (In reply to vklr@vkten.in from comment #55)
> > (In reply to Luke Kenneth Casson Leighton from comment #49)
> > > (In reply to vklr@vkten.in from comment #48)
> > 
> >        location /publicinbox/ {
> >                  access_log /var/log/nginx/public-inbox-httpd_access.log;
> >                  error_log /var/log/nginx/public-inbox-httpd_error.log;
> >                  proxy_pass         https://127.0.0.1:8080/;
> >                  proxy_buffering off;
> >          }
> 
> this fails not in nginx but in public-inbox-httpd.  the URL is passed in
> full including the word "publicinbox" and it cannot cope, it starts looking
> for a list named "publicinbox".
> 
> a rewrite rule or double proxypass is needed.
> 
> this is why i went with the domain name
> 
> > Also please install libplack-middleware-reverseproxy-perl
> > It's needed with nginx.
> 
> i have that.

I tested locally it is working. Please be careful with location "/" endings.

Anyway if this feature is not needed then?