Re: [BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On Wed, Dec 6, 2017 at 11:31 AM, Zeshy <ml.zsh@xxxxxx> wrote:
> Hi all!
>
> In search of the reason why zsh (5.4.2 and fairly recent git) is
> so much slower reading/writing larger history files (13 MB, 330k lines)
> than bash (4.4.12), I think I found a bug. If I missed something (like
> some option), and this is not a bug, but just misconfiguration, or if
> you need more information, please let me know.

This happens because of HISTSIZE and the various HIST_*DUP* options.
The only way to do duplicate filtering is to examine the entire past
history, and there's no setting of HISTSIZE that is interpreted as
"unlimited" so rewriting to output only the most recent $HISTSIZE
lines is always performed.

It would probably be possible to optimize for the case of "$HISTSIZE
is larger than the current number of lines and no duplicate removal is
enabled" but right now there is just one rewriting algorithm that
applies (or not) those conditions internal to the loop.

So not technically a bug but not optimized for people who can "afford"
the memory cost to have every shell preserve huge amounts of history.