Re: BUG: Zsh loses history entries since 2015

[Michael Stapelberg <stapelberg+zsh@xxxxxxxxxx>]

Hey Bart

Thanks for taking a look. Answers inline:

On Wed, 26 Mar 2025 at 21:21, Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx> wrote:
>
> On Tue, Mar 25, 2025 at 3:15 AM Michael Stapelberg
> <stapelberg+zsh@xxxxxxxxxx> wrote:
> >
> > Fundamentally, I think the savehistfile() function, when called because
> > Zsh is exiting, is not handling an interrupted readhistfile() call correctly.
>
> Outside of your test build, is this something that happens for example
> when shutting down a window manager or desktop, so that (possibly
> multiple) shells are being "asked" to quit?

Yes. During my work day, I open many SSH connections from my laptop
to my workstation. At the end of the work day, I kill the connection,
which means all those zsh processes will be killed.

As I said, I have run into this issue many times over the years,
it just took a lot of time to properly track it down to this bug.

> Presumably you do keep HIST_SAVE_BY_COPY set?

You can find my full zshrc at
https://github.com/stapelberg/configfiles/blob/master/zshrc
Correct, I do not touch HIST_SAVE_BY_COPY.

> > I would have attached a patch with a suggested fix, but I am not entirely sure
> > which way we should go: Would we want to gracefully handle an interrupted
> > readhistfile()? Should readhistfile() even be interruptible at all?
>
> Some users (you appear possibly to be one of them) maintain very large

I move my history files out of the way every couple of years to keep them lean.
Currently, I have 60k lines in .zsh_history (3.7 MB), and >>350k lines
in historical ~/.zsh_history_<date> files (15 MB), which zsh ignores,
but I grep on occasion when I need to go back very far in time.

I’m not sure if this qualifies as “very large” :)

> history files, so it would be impractical for it to try to block
> interrupts.  The code from workers/34817 is an attempt to handle more
> gracefully something that was already possible, though I no longer
> recall what the previous side-effects of interrupt were other than the
> parse error warning mentioned in the email thread.  The other side of
> this is that readhistfile() could be occurring at various different
> points -- particularly shell startup as well as shutdown, but also due
> to an "fc" command, which you wouldn't want to be interrupt-proof I
> suspect.

Why? What does it even mean to interrupt the loading of history at
shell startup?
Why would I ever want a shell process to only partially load my history?

I would be happy to just have history loading/saving never interrupted.

> I can think of a few possible approaches, none of which seem ideal,
> and I have not looked into implementation issues, for example that
> these should likely occur only during startup or shutodwn and "fc"
> would need another approach:
>
> 1) If reading history is interrupted, disable saving history entirely,
> perhaps by unset/reset of HISTFILE and/or SAVEHIST.  This would
> preserve file contents but lose current shell history.

Do you mean just when exiting (that seems okay) or also on startup?
Applying this behavior at startup would mean that users could run a shell
that just silently does not save to history, right? That does not sound good.

> 2) Change behavior to append current shell history to the file as if
> APPEND_HISTORY, without attempting to enforce SAVEHIST size or
> duplicate expiration.  This is probably the option most likely to

This approach would no longer rewrite history when exiting,
so at which points would zsh rewrite history then? Or would history
files only ever grow and effectively SAVEHIST size has no more effect?

> avoid losing anything, but could result in very large files.  Also, if
> you're getting interrupts during readhistfile(), what about interrupts
> during savehistfile()?

I have not checked what happens when savehistfile() is interrupted,
but yes, any fix should probably cover both readhistfile() and savehistfile().

> 3) Stop reading, but go on saving internal history per user
> configuration.  Keeps all the current shell history but would lose
> part of the existing history file.  Is that what happens now?

I think so.

To be clear: the zsh processes that are killed in my scenario do not
have any pending history to save — they just do the routine read+write
history code path on exiting, and lose part of the existing history file.

Best regards
Michael