Re: The HIST_EXPIRE_DUPS_FIRST might corrupt and wipe partially history file if many shells exit at the same time

[Piotr Karbowski <piotr.karbowski@xxxxxxxxxxxxx>]

Hi,

On 19/03/2023 17.53, Bart Schaefer wrote:
 > On Sun, Mar 19, 2023 at 4:56 AM Piotr Karbowski
 > <piotr.karbowski@xxxxxxxxxxxxx> wrote:
 >>
 >> They way I can reproduce it sometimes is if multiple zsh shells exit at
 >> the very same time. It happens when I terminate tmux session that have
 >> 10+ zsh instances,or when I just reboot my system while I have dozens of
 >> urxvt instances open.
 >>
 >>       +SAVEHIST='96000'
 >>       +setopt hist_expire_dups_first
 >>
 >> I have disabled this feature since due those corruptions. Would love to
 >> get back to it though, perhaps adding some locking mechanism would help
 >> here?
 >
 > There is a locking mechanism.  A couple of things may be happening:
 >
 > 1) Your home directory (or ZDOTDIR) is on a remote filesystem with
 > asynchronous mounting, so the locking mechanism doesn't always work.
 > You haven't described your system configuration in any detail, but
 > from what you have said this one seems unlikely.
 > 2) The locking mechanism doesn't work for some other reason.  Make
 > sure the HIST_SAVE_BY_COPY option is set (it's the default but check
 > anyway).
 > 3) When exiting a session (or particularly when rebooting), there may
 > be a limited amount of time for the processes to clean up and exit
 > before they are killed.  With the combination of options you have,
 > each shell must re-read the history file to merge it with its local
 > history before writing it back.  With multiple shells and tens of
 > thousands of lines of history, this might take longer than all the
 > shells are allowed to stay alive.
 >
 > Try putting something in your .zlogout file that writes a timestamp to
 > a file (different for each shell, e.g., use $$ in the filename).
 > Since .zlogout runs after history is saved, if you find any of those
 > files to be missing, that's a clue that #3 is happening.  You could
 > also look to see if a ".zsh_history.new" file is left behind.
Thanks for the response and applogies for missing details.

I am runnin XFS (on the top of lvm, on the top of dmcrypt).

The current setopt is (without the expire dups first)

     autopushd
     extendedglob
     noflowcontrol
     histignoredups
     histignorespace
     interactive
     interactivecomments
     promptsubst
     pushdignoredups
     pushdminus
     shinstdin

The HIST_SAVE_BY_COPY does not seems to be enabled and I am unable to
enable it. Running

     setopt HIST_SAVE_BY_COPY

Does not bring this option into list I can see with running setopt. This
is zsh-5.9. Running unsetopt without params does show nohistsavebycopy
there, however running

     setopt histsavebycopy

Does nothing, it still remains as nohistsavebycopy in unsetopt output.

The question then is -- why this feature remains disabled and cannot be
toggled on? Even running zsh -dfi to get a shell without zshrc behave
the same way, this feature remains disabled and cannot be toggled on.

-- Piotr.