comments toward completion system docs

[Roman Neuhauser <neuhauser@xxxxxxxxxx>]

Hello,

I've had hard time with the completion system documentation, and have
a few comments. The user guide requests comments be sent to workers@,
I hope the larger audience of users@ doesn't mind. This message is quite
long but easy reading, I hope it doesn't fall on deaf ears.

The text below deals with the reference manual; I had further comments
toward user guide chapter 6, but this mail grew too long. Maybe another
(shorter) email.

Oh, BTW, should the mail sound confrontational, rude or something,
please attribute that to my poor ESL, and my trying to keep this
reasonably short. I'm really just trying to be to the point. Thx!

I've been a happy Zsh user for the last couple of years, and, while a
programmer, never really "looked inside". I use only a handful of Zsh's
interactive features and never written a script in it (I'm competent in
POSIX sh scripting, though). Last week I finally decided to write
completion for a utility I've been using, so I did what I always do in
such situations, opened the manual.

Completely green as far as the completion goes (I stuck
"autoload -U compinit; compinit" in my .zshrc a few years ago and was
done with it, silently enjoying the comfort ever since), I started by
reading the completion system section, and immediately got confused. The
user guide didn't help either. I didn't know what was wrong for a couple
of days spent between the manual (18., 19.) and the guide (6.). Then it
occurred to me that the problem is strange (for me) organization and
failure to present the larger concepts or put the documented features
into larger context, in a somehow structured way.

What I needed first was a clear description of the big picture,
and I didn't get it. The manual immediately starts drilling little
(at that point premature) details, without saying what it's talking
about until much later, deep in the text.

This imperfection is present throughout the completion sections on two
scales.

The smaller one that's easier to fix is incomplete synopses.  These
omissions aren't fatal, but do require the reader to reconstruct
the structure from description, provided it's thorough enough to cover
all possibilities. An example of this, the _arguments function, is
discussed a paragraph further.

The larger variant is (FMPOV) frequent putting of the horse before the
carriage, where the material seems to be presented inorder instead of
preorder: the reader isn't offered the higher level informational glue
until they read about much of (and (and (and lower) lower) lower) level
stuff which, without the glue, is of limited value to the reader (and
so harder to remember).  18. Completion Widgets comes before
19. Completion System.

I tried to present the larger scale version following the _arguments
case, using 19.1 and beginning of 19.2 as an example. My version of the
text reflects my current understanding of the topic.  Any bullshit is,
of course, all mine.

: _arguments [ -swWACRS ] [ -O name ] [ -M matchspec ] [ : ] spec ...
: 
: This function can be used to give a complete specification for
: completion for a command whose arguments follow standard UNIX option
: and argument conventions. The following forms specify individual sets
: of options and arguments; to avoid ambiguity, these may be separated
: from the options to _arguments itself by a single colon.
: 
: n:message:action
: n::message:action

The problem here is that the nonterminals aren't really specified.
This case isn't that bad, one can easily deduce that those
*:message:action nonterminals are branches of spec, but the optspec that
follows is a bit tougher nut to crack:

: optspec
: optspec:...
: 
: This describes an option. The colon indicates handling for one or more
: arguments to the option; if it is not present, the option is assumed to
: take no arguments.
 
: Each optarg following an optspec must take one of the following forms:
: 
: :message:action
: ::message:action

Does that mean optarg should come as the next word ("optspec: optarg"),
or is it "optspec:optarg"? And if it's the latter, would an option that
accepts a single mandatory argument be "-x[m]::m:a" or "-x[m]:m:a"?
IOW, is the colon in "optspec:..." that first colon from the argument,
or not? It doesn't say, until several pages of text, at the very end of
the _arguments description an example gives the answer... But examples
should illustrate, not replace, documentation, no?

: 19.1 Description
:
: This describes the shell code for the new completion system. It
: consists of various shell functions; those beginning `comp' are to be
: called directly, while those beginning `_' are called by the completion
: code. The shell functions of the second set, which implement completion
: behaviour and may be bound to keystrokes, are referred to as `widgets'. 

This isn't description of the completion system. To deserve such a title
it would need to describe what completion is ("completion"), and how is
it organized ("system"). Perhaps something like this would be attacking
the topic at a more systematic way:

# Completion is a feature present in many shells. It allows the user
# to type only a part (usually the prefix) of a word and have the shell
# fill in the rest. Zsh's completion system is programmable, the shell
# can be for example set to complete email addresses in arguments to the
# mail command from your ~/.abook/addressbook; usernames, hostnames, and
# even remote paths in arguments to scp, and so on. Anything that can be
# written in or glued together with zsh can be the source of what the
# ZLE offers when asked for possible completions.
# 
# Zsh has two completion systems, an old, so called compctl completion
# (named after the builtin command that serves as its complete and only
# user interface), and a new one, called simply (new) completion system,
# organized as library of builtin and user-defined functions.
#
# The two systems differ in their interface for specifying the completion
# behavior, and also in that the new completion system (TBD) while
# the old system (TBD).

Something like that should come as the /first/ thing regarding
completion, that is, before sections 18, 19, and 20.

The above quote is a full text of 19.1 in the current manual, which then
immediately proceeds to discuss compinit, compinstall, zstyle:

: 19.2 Initialization
:
: If the system was installed completely, it should be enough to call the
: shell function compinit (...)

That's way too early given that the reader still has no idea what he's
looking at.

Below is a rough sketch attempt at a more structured approach. It's
likely to contain factual errors; I'm just trying to hint at the
structure. Note however that it is crucial that the short introductions
don't grow beyond one to three paras of very ontopic, properly scoped
information per concept introduced.

# The "new" completion system accepts behavior specifications in terms
# of functions (but see compdef). The behavior map is keyed by
# so called /contexts/, which describe what is being completed in one of
# two abstract ways:
# 
# * "normal" /contexts/, these describe the position on the command line
#   at which completion is requested ("first argument to rmdir, the word
#   being completed names a directory"), and
# * "special" /contexts/, these denote certain elements in Zsh's syntax:
#   "a word in command position" or "an array subscript"...
#
# Besides commands names and contexts, the system employs two more
# concepts, /styles/ and /tags/, to augment the behaviors.  These are
# modifiers or overrides for normal behaviors of the functions
# implementing completion.
#
# Tags play a dual role. One, they serve as a classification system for
# the matches (e. g. completion for the ls command can prefer to try
# files before directories). Second, they are the rightmost (least
# significant) element in a context specification.
#
# Tags are a form of parameterization for the behaviors aimed at the
# "what" of completion.
# 
# (TBD: more tags description)
#
# Styles modify various operations of the completion system, such as
# output formatting, but also what kinds of completers are used (and in
# what order), or which tags are examined. Styles may accept arguments.
#
# Styles are a form of parameterization for the behaviors aimed more at
# the "how" of completion.
#
# At various points of execution, the completion system checks what
# styles and/or tags are defined for the current context, and uses that
# to modify its behavior.
#
# (TBD...)
#
# When a completion is requested, a dispatcher function is called
# (see _main_complete). This dispatcher decides which function should
# be called to produce the completions, and calls it. The result is
# passed to one or more /completers/, functions that implement
# individual completion strategies: simple completion, error correction,
# completion with error correction, menu selection, etc.
#
# (TBD...)

-- 
How many Vietnam vets does it take to screw in a light bulb?
You don't know, man.  You don't KNOW.
Cause you weren't THERE.             http://bash.org/?255991