Re: archived messages with "From " get truncated

["Daniel Shahaf" <d.s@xxxxxxxxxxxxxxxxxx>]

Vincent Lefevre wrote on Thu, 24 Jun 2021 23:36 +00:00:
> On 2021-06-24 19:11:56 +0000, Daniel Shahaf wrote:
> > Vincent Lefevre wrote on Thu, Jun 24, 2021 at 10:36:12 +0200:
> > > Below is a message that was sent by Stephane to workers.
> > > It has been archived here:
> > > 
> > >   https://www.zsh.org/mla/workers/2021/msg01272.html
> > > 
> > > but got truncated just before a line starting with "From ".
> > > It seems that the mail archive software is buggy, thinking
> > > that this starts a new mail message.
> > 
> > The problem might be either in the mailing list software or in how we
> > invoke it.
> > 
> > We invoke mhonarc as follows:
> > 
> >    134          /usr/local/bin/mhonarc \
> >    135                  -quiet \
> >    136                  -definevar listlocalpartsansprefix=${listlocalpart#zsh-} \
> >    137                  -title  "${title}" \
> >    138                  -ttitle "${title}" \
> >    139                  -tlevels 9999 \
> >    140                  -rcfile /usr/local/www/mhonarc.zsh \
> >    141                  -add \
> >    142                  -- "$munged_tmpfile" \
> >    143            || exit EX_UNAVAILABLE
> > 
> > Here, ${munged_tmpfile} is a file that was created by
> > «() { munged_tmpfile=$1; cat > $munged_tmpfile } =(:)», the whole thing
> > being invoked by Exim using a «"| /path/to/script"» target in
> > /etc/aliases.  (The "munged" terminoilogy is because there's also
> > a «perl -pi -E 's/^X-Seq: …/…/ if (1../^$/)'» in there.)
> > 
> > Does anyone happen to see off the top of their heads what we're doing
> > wrong?
> 
> If I understand correctly, $munged_tmpfile is a *single* message.

Indeed.

> Then, according to the mhonarc documentation
> 
>   https://www.mhonarc.org/MHonArc/doc/quickstart.html#adding
> 
> you must not use the "-add" option, but "-single".

Thanks!  I missed that flag.  I'll have a look.  (If someone can beat me
to it, please do.)

> Alternatively,
> you may use "-add" for a single message, but the message must be
> read from the standard input instead of being a filename argument.

Yup.  We actually already have a pending merge request for this, but it
fell through the cracks ☹

(MR !5, issue #37; not public, sorry).

> The examples:
>   (1) mhonarc -add <path>/mailfolder
>   (2) mhonarc -add < single.msg
>   (3) cat single.msg | mhonarc -add
>   (4) mhonarc -single < messagefile > file.html
>   (5) mhonarc -single messagefile > file.html
> 
> The syntax above is case (1), i.e. with a folder. And I suppose
> that mhonarc regards a line starting with "From " as the beginning
> of a new mail message from the folder, even if this line has a
> wrong syntax for such a mail separator. Hence the truncation.
> Then, I don't know what happens with the remaining part as this
> would yield an invalid mail message; perhaps it is just discarded.

Apparently it's discarded, unless it looks like a valid email message,
as in the case of git-format-patch(1) emails; see workers/48587 (which
is https://www.zsh.org/mla/workers/2021/msg00812.html).

And yes, I had to give the full link, because the numlist file (= the
database for the "X-Seq number to HTML" redirection script) is broken ☹

So, action items:

1. Review «mhonarc -single»
2. Deploy either «mhonarc -single» or «mhonarc -add < $munged_tmpfile»
3. Regenerate broken messages in the archives [for all lists]
4. Fix appends to the numlist file
5. Regenerate the missing part of the numlist file [for all lists]

Anyone interesting in helping, don't be shy ☺

Thanks a lot, Vincent.

Daniel