Re: Memory usage of history?

["Nikolay Aleksandrovich Pavlov (ZyX)" <kp-pav@xxxxxxxxx>]

27.06.2016, 02:30, "Bart Schaefer" <schaefer@xxxxxxxxxxxxxxxx>:
> On Jun 25, 8:33pm, Nikolay Aleksandrovich Pavlov (ZyX) wrote:
> } Subject: Re: Memory usage of history?
> }
> } 1. allows saving user-defined metada in history file and
>
> I'm not sure the answer to the history file being too large is to
> make it even larger by cramming in all sorts of other data. This
> would be even slower to parse at load time as well.

The idea is that history file needs not be too large, but without more advanced criteria using big SAVEHIST value is needed to not miss useful, but uncommon entries. So adding metadata will reduce history size not because metadata reduces history entry size, but because smaller SAVEHIST is needed.

>
> } 2. allows user to get control over what exactly will be removed.
>
> In addition to all the other stuff I mentioned, I forgot about the
> relatively recent addition of the HISTORY_IGNORE variable, which can
> be a pattern that matches lines to leave out. That would be the
> best way to handle my "foo is a throwaway" and similar criteria.

I cannot say I have any such patterns.

>
> } Specifically first may be used to save information about
> }
> } 1. How often the command is used (total number of uses, anything else like
> } "uses per month" would be harder to determine).
> } 2. Time it took command to type (when it was typed for the first time)
> } (time between first self-insert (or $*BUFFER modification if it was
> } constructed by a widget) and accept-line).
> } 3. Last time command was run.
> } 4. Time it took command to finish (average among all runs).
> } 5. What was the exit code (hash exit code - number of times it occurred).
>
> I find these to be very unlikely criteria for deciding what's interesting
> in the history?
>
> For one thing, "time it took to type" is going to be really hard to get
> right; multi-line commands have multiple accept-line calls, and you'd
> have to filter out commands that were recalled from the history or you'd
> get an average much too small.

Filtering out commands that were recalled from history is not hard: there are not too much widgets that do this. Though it may be practical for commands that were recalled from history and modified to add “time it took to type” from the original command to the modified one.

Also some prompt %format allows determining whether command is a continuation, so saving previous time and adding it on next accept-line is not a problem.

>
> Larger number of uses would be skewed towards really simple things, and
> in fact (at least in my own case) the LESS often I use a command, the
> more likely I am to want it from the history (unless it's one of those
> throwaways I mentioned in another message), because I can remember the
> ones I use a lot without zsh's help. If I use it often enough, I can
> make an alias or keybinding for it and not need to search history.

My main point was making a custom function that allows to adjust criteria. I suggested this because I tend to keep in history some commands which are rather easy to retype, but I need them fast, and I do not want to have 100500 aliases for easy commands in my zshrc.

>
> How long the command took to run seems entirely unrelated to whether
> it is history-worthy (and also doesn't work with shared/incremental
> history). What would you use the exit code for, except maybe weeding
> out typos?

Exit code is for typos, and it was put on the last place because it is not much useful for other purposes. How long it took command to run is second-least-useful, but still has something to do with history: usually I do not want to repeat long-running commands, if needed they are first candidates to be run using `screen`/`tmux`/… which will be another history entry.

>
> I like Christian Neukirchen's idea of maintaining a daily archive.
> Adding a function / keybinding to search through an alternate history
> store seems more manageable than either having a huge history always
> in memory or a complicated AI for storing only interesting bits.

So far I am fine with my variant “just use large SAVEHIST and HISTSIZE”. Just suggested a way to reduce the number of entries that need to be stored that came to my mind.