terminal queries (was Re: PATCH: changing terminal cursor form)

[Oliver Kiddle <opk@xxxxxxx>]

Attached is a patch which goes on top of my various terminal extension
patches from early in the year that clears it up to a state where I'd
like to start committing the changes. So if you have any further
comments, speak out.

SIGINT/Ctrl-C does now interrupt the querywait interval. Taking a fresh
look after the long break helped and that was easy this time.

One other notable difference is that querying for kitty keyboard
extensions is disabled by default. To be useful, that needs a layer for
translating keys so that all the default key bindings work.

Aside from that, I've mostly tried to address Mikael's concerns:

On 3 Apr, Mikael Magnusson wrote:
> This feels a bit confusing to me, the name suggests it is an array
> containing enabled extensions, but the above suggests that an empty

I've substantially redone the documentation here.

> Actually after rereading the documentation I am even more confused,
> there is "query-cursor", but no "cursor", only "cursor-color" and
> "cursor-shape", but query-cursor only queries the color, and enables
> both of these features? Meanwhile the color is stored in .term.cursor,
> not .term.cursorcolor, and is customized via zle_cursorform. Maybe
> some consistency with the naming would be helpful here.

I've renamed the query extension to query-cursorcolor. And added
documentation to spell out that many terminal queries just populate
variables and don't do feature detection.

> I've never seen the word cursor form before, I assume it is supposed
> to entail the combination of shape and color? I couldn't see the term
> explicitly introduced anywhere.

I can't think of anything better for encapsulating both shape and color
even if "form" would ususally only cover shape. Rather than define the
term, I've reworded to never use it as a term. It does persist in the
variable name, however. And in the source code.

> I realized there's another problem with this scheme, the above code is
> invalid on older zsh versions producing errors like:
> % .term.extensions+=( -cursor-shape )
> zsh: unknown file attribute:
> which makes it harder to share your config across hosts (wrapping
> things in version checks is pretty cumbersome imo).

> This problem would be somewhat lessened if we release a version with
> namespaces first, and then add mechanisms that use them in the main
> shell in a subsequent release.

I don't see how releasing with namespaces first helps. Unless you want
to wait until that release has percolated through to the oldest systems
still in wide use which would be more than a decade. For highlight
groups (which also use namespaces), I'm relying on the fact that they'll
land in the same release as namerefs. It's easy enough to test for
features or redirect to /dev/null. For example, you can define:

  function have.namespaces=1 { false }
  have.namespaces=1 && .term.extensions+=( ... )

> That aside, I was also wondering if there is a 'context' for programs,
> eg is there a mechanism for automatically changing the cursor form
> when an external program is running and then when returning to the
> shell again? I feel like it would be a bit weird if programs got a
> different cursor depending on which vi/emacs/insert/overwrite mode you
> were in when you started it

It resets the prompt before running a command so they'll get the
default. Additional contexts or shell hooks can be added in future. The
main aim here was to cover vi mode which many vi users want.

> > You'll be unaffected other than getting an underline cursor for overwrite mode.

> I can already say I would hate that, and based on the feedback we got

The patch drops that default, leaving only vi insert and pending modes
having a default.

Oliver

diff --git a/Doc/Zsh/params.yo b/Doc/Zsh/params.yo
index 41c7cae87..34825deae 100644
--- a/Doc/Zsh/params.yo
+++ b/Doc/Zsh/params.yo
@@ -965,6 +965,12 @@ there is a block of reserved or unused signal numbers before the POSIX
 real-time signals so the array index can't be used as an accurate indicator
 of their signal number. Use, for example, tt(kill -l SIGRTMIN) instead.
 )
+vindex(.term.cursor)
+item(tt(.term.cursor))(
+The default color of the cursor.
+
+See also the tt(.term.extensions) parameter.
+)
 vindex(.term.bg)
 item(tt(.term.bg))(
 The background color of the terminal if the terminal supports the
@@ -1731,80 +1737,8 @@ arrays this is not tied to a zsh array.
 )
 vindex(.term.extensions)
 item(tt(.term.extensions))(
-An array containing a list of extension features of the terminal to be enabled
-or disabled (prefixed with `tt(-)').  When ZLE starts, it will add entries for
-features that were auto-detected.  This auto-detection uses extensions itself,
-all named with a `tt(query)' prefix.  As these are used when ZLE starts they
-would need to be disabled early in the startup files if they are to be
-disabled.  A value of `tt(-query)' will disable all terminal queries on
-startup.  Extensions can be any of the following, where the marks `<D>' and
-`<E>' indicate whether they are disabled or enabled by default:
-
-startitem()
-item(tt(bracketed-paste) <E>)(
-Many terminal emulators have a feature that allows applications to identify
-when text is pasted into the terminal rather than being typed normally.  For
-ZLE, this means that special characters such as tabs and newlines can be
-inserted instead of invoking editor commands. Furthermore, pasted text forms a
-single undo event and if the region is active, pasted text will replace the
-region.
-)
-item(tt(cursor-color) <E>)(
-Support for changing the color of the cursor.
-)
-item(tt(cursor-shape) <E>)(
-Support for changing the shape of the cursor.
-)
-item(tt(integration-output) <E>)(
-This provides the terminal with semantic information regarding where the output
-from commands start and finish. Some terminals use this information to make it
-easy to select just the output from a single command.
-)
-item(tt(integration-prompt) <E>)(
-This informs the terminal when the shell displays a prompt. This enables a
-variety of terminal features such as displaying markers, allowing you to jump
-to previous commands in the scroll-back buffer and ensuring that right prompts
-are handled correctly when the window is resized. The end of the prompt also
-provides the terminal with a marker for the start of user input.
-)
-item(tt(integration-pwd) <E>)(
-This advises the terminal of the shell's current directory which allows it
-to create new windows with the same current working directory.
-)
-item(tt(query-bg) <E>)(
-Query the terminal background color which is used for tt(.term.bg) and
-tt(.term.mode).
-)
-item(tt(query-cursor) <E>)(
-Query the cursor color. This facilitates restoring the cursor to its original
-color if it has been configured via tt(zle_cursorform). The color is also
-assigned to tt(.term.cursor).
-)
-item(tt(query-fg) <E>)(
-Query the terminal foreground color which is used for tt(.term.fg).
-)
-item(tt(query-id) <E>)(
-Query the terminal identification which is used for tt(.term.id) and
-tt(.term.version).
-)
-xitem(tt(modkeys-kitty) <D>)
-item(tt(query-modkeys-kitty) <E>)(
-Support for the kitty keyboard handling protocol which enables reporting of a
-wider range of key combinations and resolves problems with ambiguous key
-sequences.  Currently there is only support for detecting whether the terminal
-supports this feature.
-)
-item(tt(modkeys-xterm) <D>)(
-Support for the keyboard handling sequences associated with xterm's
-tt(modifyOtherKeys) X resource. This enables reporting of a wider range of key
-combinations and resolves some problems with ambiguous key sequences.
-)
-xitem(tt(truecolor) <D>)
-item(tt(query-truecolor) <E>)(
-Support for 24-bit truecolor escape sequences.  Auto-detection also tries
-termcap and the tt(COLORTERM) environment variable.
-)
-enditem()
+An array containing a list of extension features of the terminal.
+See sectref(Terminal Extensions)(zshzle).
 )
 vindex(.term.querywait)
 item(tt(.term.querywait))(
@@ -1922,8 +1856,8 @@ vindex(zle_cursorform)
 item(tt(zle_cursorform))(
 An array describing contexts in which ZLE should change the shape and color
 of the cursor.
-See ifzman(em(Cursor Form) in zmanref(zshzle))\
-ifnzman(noderef(Cursor Form)).
+See ifzman(em(Cursor Shape and Color) in zmanref(zshzle))\
+ifnzman(noderef(Cursor Shape and Color)).
 )
 vindex(zle_highlight)
 item(tt(zle_highlight))(
diff --git a/Doc/Zsh/zle.yo b/Doc/Zsh/zle.yo
index 669e6927b..e20184121 100644
--- a/Doc/Zsh/zle.yo
+++ b/Doc/Zsh/zle.yo
@@ -54,6 +54,7 @@ menu(Zle Builtins)
 menu(Zle Widgets)
 menu(User-Defined Widgets)
 menu(Standard Widgets)
+menu(Terminal Extensions)
 menu(Character Highlighting)
 endmenu()
 
@@ -1196,7 +1197,7 @@ This can be used for detecting switches between the vi command
 )
 enditem()
 
-texinode(Standard Widgets)(Character Highlighting)(User-Defined Widgets)(Zsh Line Editor)
+texinode(Standard Widgets)(Terminal Extensions)(User-Defined Widgets)(Zsh Line Editor)
 sect(Standard Widgets)
 cindex(widgets, standard)
 The following is a list of all the standard widgets,
@@ -2612,7 +2613,102 @@ argument, multiple words will be selected.
 )
 enditem()
 
-texinode(Character Highlighting)()(Standard Widgets)(Zsh Line Editor)
+texinode(Terminal Extensions)(Character Highlighting)(Standard Widgets)(Zsh Line Editor)
+sect(Terminal Extensions)
+
+vindex(.term.extensions, setting)
+Modern terminals often carry functionality that is not covered by the
+`terminfo' library.  For these cases, the availability of a feature can be
+asserted by including it in the array parameter tt(.term.extensions).
+Similarly the absence of a feature can be indicated via it's inclusion with a
+`tt(-)' prefix.  Where a feature isn't listed, the status of terminal support
+is unknown and defaults apply as to whether ZLE will use the feature.  This
+is often harmless as terminals ignore extra instructions they don't recognise.
+Even if your terminal does support a particular extension, you may choose to
+list it with an initial `tt(-)' as a way to disable the associated feature.
+Many extensions are named with common prefixes to group related extensions.
+This allows a whole class of extensions to be disabled with a single entry such
+as by adding `tt(-cursor)' to disable cursor shape and color changing.
+
+When ZLE starts, it will add entries for features that were auto-detected.  This
+auto-detection uses extensions itself, all named with a `tt(query)' prefix.  As
+this happens when ZLE starts disabling them needs to be done early in the
+startup files.  A value of `tt(-query)' will disable all terminal queries on
+startup, including those that query terminal properties such as colors rather
+than detecting features.  Populating the array with the status of auto-detected
+features will mean that associated terminal queries are not needed and will be
+skipped.
+
+Extensions can be any of the following, where the marks `<D>' and `<E>'
+indicate whether each one is disabled or enabled by default:
+
+startitem()
+item(tt(bracketed-paste) <E>)(
+Many terminal emulators have a feature that allows applications to identify
+when text is pasted into the terminal rather than being typed normally.  For
+ZLE, this means that special characters such as tabs and newlines can be
+inserted instead of invoking editor commands. Furthermore, pasted text forms a
+single undo event and if the region is active, pasted text will replace the
+region.
+)
+item(tt(cursor-color) <E>)(
+Support for changing the color of the cursor.
+)
+item(tt(cursor-shape) <E>)(
+Support for changing the shape of the cursor.
+)
+item(tt(integration-output) <E>)(
+This provides the terminal with semantic information regarding where the output
+from commands start and finish. Some terminals use this information to make it
+easy to select just the output from a single command.
+)
+item(tt(integration-prompt) <E>)(
+This informs the terminal when the shell displays a prompt. This enables a
+variety of terminal features such as displaying markers, allowing you to jump
+to previous commands in the scroll-back buffer and ensuring that right prompts
+are handled correctly when the window is resized. The end of the prompt also
+provides the terminal with a marker for the start of user input.
+)
+item(tt(integration-pwd) <E>)(
+This advises the terminal of the shell's current directory which allows it
+to create new windows with the same current working directory.
+)
+item(tt(query-bg) <E>)(
+Query the terminal background color which is used for tt(.term.bg) and
+tt(.term.mode).
+)
+item(tt(query-cursorcolor) <E>)(
+Query the cursor color. This facilitates restoring the cursor to its original
+color if it has been configured via tt(zle_cursorform). The color is also
+assigned to tt(.term.cursor).
+)
+item(tt(query-fg) <E>)(
+Query the terminal foreground color which is used for tt(.term.fg).
+)
+item(tt(query-id) <E>)(
+Query the terminal identification which is used for tt(.term.id) and
+tt(.term.version).
+)
+xitem(tt(modkeys-kitty) <D>)
+item(tt(query-modkeys-kitty) <D>)(
+Support for the kitty keyboard handling protocol which enables reporting of a
+wider range of key combinations and resolves problems with ambiguous key
+sequences.  Currently there is only support for detecting whether the terminal
+supports this feature.
+)
+item(tt(modkeys-xterm) <D>)(
+Support for the keyboard handling sequences associated with xterm's
+tt(modifyOtherKeys) X resource. This enables reporting of a wider range of key
+combinations and resolves some problems with ambiguous key sequences.
+)
+xitem(tt(truecolor) <D>)
+item(tt(query-truecolor) <E>)(
+Support for 24-bit truecolor escape sequences.  Auto-detection also tries
+termcap and the tt(COLORTERM) environment variable.
+)
+enditem()
+
+texinode(Character Highlighting)()(Terminal Extensions)(Zsh Line Editor)
 sect(Character Highlighting)
 
 vindex(zle_highlight, setting)
@@ -2855,9 +2951,10 @@ special array parameter tt(region_highlight); see
 ifnzman(noderef(Zle Widgets))\
 ifzman(above).
 
-texinode(Cursor Form)()()(Character Highlighting)
-subsect(Cursor Form)
-cindex(cursor form)
+texinode(Cursor Shape and Color)()()(Character Highlighting)
+subsect(Cursor Shape and Color)
+cindex(cursor shape)
+cindex(cursor color)
 
 vindex(zle_cursorform, setting)
 Some terminals support the ability to change the shape and color of the cursor.
@@ -2870,7 +2967,7 @@ Each element of the array should consist of a word indicating a context
 followed by a colon, then a comma-separated list of properties describing the
 shape and color to apply to the cursor.
 
-The available contexts follow with the default cursor form shown in
+The available contexts follow with the default value shown in
 parentheses. Where no default is given, the terminal's default is applied:
 
 startitem()
@@ -2884,7 +2981,7 @@ mode.
 item(tt(insert) (tt(bar)))(
 Used for vi editing mode.
 )
-item(tt(overwrite) (tt(underline)))(
+item(tt(overwrite))(
 Used when editing text in overwrite mode or with the vi replace command.
 )
 item(tt(pending) (tt(underline)))(
@@ -2897,7 +2994,8 @@ Applied for both tt(regionstart) and tt(regionend) contexts.
 item(tt(regionstart))(
 Used when the region is active and the cursor is positioned at the start of the
 region. The region includes the text under the cursor when it is positioned at
-the start so it is best to choose a cursor form that does not obscure this fact.
+the start so it is best to configure the cursor so that it does not obscure
+this fact.
 )
 item(tt(regionend))(
 Used when the region is active and the cursor is positioned at the end of the
@@ -2910,9 +3008,9 @@ cursor position so the same advice as for tt(regionstart) applies.
 )
 enditem()
 
-The available cursor forms are tt(none), tt(bar), tt(block), tt(underline) and
+The available cursor shapes are tt(none), tt(bar), tt(block), tt(underline) and
 tt(hidden). Additionally, you can specify either tt(blink) or tt(steady) to
 indicate whether the cursor should flash and specify a color as an RGB triplet
-in hexadecimal format with with tt(color=)var(#xxxxxx). The value tt(none)
-applies the terminal's default cursor form. Note that on many terminals, this
-may be different to the initial cursor state from when the shell started.
+in hexadecimal format with tt(color=)var(#xxxxxx). The value tt(none)
+applies the terminal's defaults. Note that on many terminals, this may
+differ from the initial cursor state from when the shell started.
diff --git a/Src/Zle/termquery.c b/Src/Zle/termquery.c
index 6257bd6c2..58b2edddb 100644
--- a/Src/Zle/termquery.c
+++ b/Src/Zle/termquery.c
@@ -222,7 +222,7 @@ probe_terminal(const char *tquery, seqstate_t *states,
     gettyinfo(&ti);
     memcpy(&torig, &ti, sizeof(torig));
 #ifdef HAS_TIO
-    ti.tio.c_lflag &= (~ECHO & ~ICANON & ~ISIG);
+    ti.tio.c_lflag &= (~ECHO & ~ICANON);
     ti.tio.c_iflag &= ~ICRNL;
 #else
     ti.sgttyb.sg_flags &= ~ECHO;
@@ -251,7 +251,12 @@ probe_terminal(const char *tquery, seqstate_t *states,
 		memset(current, 0, blen);
 		blen *= 2;
 	    }
-	    if ((ch = getbyte(timeout, 0, 1)) == EOF)
+            ch = getbyte(timeout, 0, 1);
+	    if (errflag) {
+		errflag = 0;
+		break;
+	    }
+	    if (ch == EOF)
 		break;
 	    *current++ = ch;
 	    illgotten = current;
@@ -451,7 +456,7 @@ handle_color(int bg, int red, int green, int blue)
 
 /* roughly corresponding feature names */
 static const char *features[] =
-	{ "bg", "fg", "cursor", "modkeys-kitty", "truecolor", "id" };
+	{ "bg", "fg", "cursorcolor", "modkeys-kitty", "truecolor", "id" };
 static const char *queries[] =
 	{ TQ_BGCOLOR, TQ_FGCOLOR, TQ_CURSOR, TQ_KITTYKB, TQ_RGB, TQ_XTVERSION, TQ_DA };
 
@@ -501,13 +506,17 @@ query_terminal(void) {
     for (i=0; i < sizeof(queries)/sizeof(*queries); i++) {
 	int last = i >= sizeof(features)/sizeof(*features);
 	int found = (last && tqend == tquery);
+	int enable = 0;
 	char *cterm;
 
 	/* skip if the query or corresponding feature is already in the list */
-	for (f = flist; !last && !found && f && *f; f++)
-	    found = !strcmp(*f + (**f == '-'), features[i]) ||
-		(!strncmp(*f, "-query-", 7) && !strcmp(*f + 7, features[i]));
-	if (found)
+	for (f = flist; !last && !found && f && *f; f++) {
+	    /* just i=3(TQ_KITTYKB) is disabled by default */
+	    enable = i == 3 && strpfx("query-", *f) && !strcmp(*f + 6, features[i]);
+	    found = enable || !strcmp(*f + (**f == '-'), features[i]) ||
+		(strpfx("-query-", *f) && !strcmp(*f + 7, features[i]));
+	}
+	if (found ? !enable : i == 3)
 	    continue;
 	/* if termcap indicates 24-bit color, assume support - even
 	 * though this is only based on the initial $TERM
@@ -713,10 +722,10 @@ prompt_markers(void)
 {
     static unsigned int aid = 0;
     static char pre[] = "\033]133;A;cl=m;aid=zZZZZZZ\033\\"; /* before the prompt */
-    static const char *const PR = "\033]133;P;k=i\033\\";   /* primary (PS1) */
-    static const char *const SE = "\033]133;P;k=s\033\\";   /* secondary (PS2) */
-    static const char *const RI = "\033]133;P;k=r\033\\";   /* right (RPS1,2) */
-    static const char *markers[] = { pre, PR, SE, RI };
+    static const char PR[] = "\033]133;P;k=i\033\\";   /* primary (PS1) */
+    static const char SE[] = "\033]133;P;k=s\033\\";   /* secondary (PS2) */
+    static const char RI[] = "\033]133;P;k=r\033\\";   /* right (RPS1,2) */
+    static const char *markers[] = { PR, SE, RI };
     static const char *nomark[] = { NULL, NULL, NULL, NULL };
 
     if (!extension_enabled("integration", "prompt", 11, 1))
@@ -844,7 +853,6 @@ zle_set_cursorform(void)
 	cursor_forms = zalloc(CURC_DEFAULT * sizeof(*cursor_forms));
     memset(cursor_forms, 0, CURC_DEFAULT * sizeof(*cursor_forms));
     cursor_forms[CURC_INSERT] = CURF_BAR;
-    cursor_forms[CURC_OVERWRITE] = CURF_UNDERLINE;
     cursor_forms[CURC_PENDING] = CURF_UNDERLINE;
 
     for (; atrs && *atrs; atrs++) {
@@ -876,7 +884,7 @@ void
 free_cursor_forms(void)
 {
     if (cursor_forms)
-	zfree(cursor_forms, CURC_DEFAULT * sizeof(*cursor_form));
+	zfree(cursor_forms, CURC_DEFAULT * sizeof(*cursor_forms));
     cursor_forms = 0;
 }