Re: Preserve DPUTS output in comptest-based tests

[Philippe Altherr <philippe.altherr@xxxxxxxxx>]

Can you at least write something about what mechanism you're trying to
implement instead of just "handle DPUTS"? For example, at a glance a
reader would wonder "If there is spurious DPUTS output, surely this
will already fail the test because the test expects a certain output"?
Maybe you could mention that some tests discarded zpty output without
reading it, if that's indeed what is happening?

For the record this all started because I added a print statement in makecompparams and discovered to my great surprise that apparently not a single test was triggering it. After some digging, I realized that most of the output of comptest based tests is simply ignored. If you add a DPUTS in makecompparams, not a single test fails. With my patch all 6 Y tests fail. If you add a DPUTS at the start of zle_main_entry, only 1 X and 1 Y test fail. With my patch, 4 X tests and all 6 Y tests fail.

Only 3 X and all 6 Y tests use comptest. Not a single one of them reads any of the log parameters initialized in comptest with zpty output. In other words, zpty output read in comptest is currently only processed in comptest. That processing currently ignores the vast majority of DPUTS messages. Afaict, it ignores most of the output and only looks for a few specific tokens.

> -  zpty -r zsh log1 "*<PROMPT>*" || {
> +  zpty_read log1 "*<PROMPT>*" || {
>      print "first prompt hasn't appeared."
>      return 1
>    }

this zpty call didn't use -m before, was that a bug? If so, you should
probably mention it in the commit message, if not, mention why adding
it is not a problem.

The -m flag tells zpty to return 0 iff the pattern (here "*<PROMPT>*") is matched. Without -m, it also returns 0 if the command exited and some output was read. The lack of -m looks like a bug to me. The same checks for "*<PROMPT>*" in zpty_run and comptesteval use -m. Adding -m doesn't break any existing test.

I have added documentation to the two new functions in the updated patch below.

- Preserve DPUTS output in comptest-based tests

One thing I don't like too much with the current patch is that it doesn't respect the output order. My first attempt to solve the issue was to only modify the comptest function as shown in the second patch. If you apply it on top of the first one and run the test Y01completion.ztst, you get the following output:

./Y01completion.ztst: starting.

--- /tmp/zsh.ztst.1792/ztst.out 2026-06-18 02:18:45

+++ /tmp/zsh.ztst.1792/ztst.tout 2026-06-18 02:18:46

@@ -1,10 +1,12 @@

 line: {: }{}

+DPUTS:{ complete.c:1342: !!! makecompparams !!!}

 DESCRIPTION:{file}

 DI:{dir1}

 DI:{dir2}

 FI:{file1}

 FI:{file2}

 line: {: dir1/}{}

+DPUTS:{ complete.c:1342: !!! makecompparams !!!}

 line: {: dir2/}{}

 line: {: file1}{}

 line: {: file2}{}

Test ./Y01completion.ztst failed: output differs from expected as shown above for:

  comptest $': \t\t\t\t\t\t\t'

Was testing: directories and files

./Y01completion.ztst: test failed.

The DPUTS messages appear in between other messages, presumably in the order they were found in the output.

If you comment out the return statement in zpty_handle_dputs added by the second patch and rerun the same test, you get the following output, which is also what you get with just my first patch:

./Y01completion.ztst: starting.

--- /tmp/zsh.ztst.2138/ztst.out 2026-06-18 02:19:48

+++ /tmp/zsh.ztst.2138/ztst.tout 2026-06-18 02:19:48

@@ -1,3 +1,5 @@

+DPUTS:{ complete.c:1342: !!! makecompparams !!!}

+DPUTS:{ complete.c:1342: !!! makecompparams !!!}

 line: {: }{}

 DESCRIPTION:{file}

 DI:{dir1}

Test ./Y01completion.ztst failed: output differs from expected as shown above for:

  comptest $': \t\t\t\t\t\t\t'

Was testing: directories and files

./Y01completion.ztst: test failed.

Here the DPUTS messages are all grouped at the top.

Unfortunately my first attempt isn't general enough; it's still ignoring many DPUTS messages. That's why I opted for the more general solution implemented in the proposed patch. It's definitely better than ignoring DPUTS messages. However, the fact that the DPUTS messages may be printed out of order with respect to other messages could be confusing. I'm wondering whether tackling this ordering issue would be worth the added complexity.

Philippe

diff --git a/Test/comptest b/Test/comptest
index 39ad14768..c006c3efc 100644
--- a/Test/comptest
+++ b/Test/comptest
@@ -21,7 +21,7 @@ comptestinit () {
   export PS1="<PROMPT>"
   zpty zsh "$comptest_zsh -f +Z"
 
-  zpty -r zsh log1 "*<PROMPT>*" || { 
+  zpty_read log1 "*<PROMPT>*" || {
     print "first prompt hasn't appeared."
     return 1
   }
@@ -99,20 +99,47 @@ bindkey -a "^X" zle-finish
 '
 }
 
+# Reads output until the pattern $2 is matched or the end is reached.
+# Detects, prints, and removes any DPUTS messages found in the output.
+# Stores the result in the parameter named $1. Returns 0 iff the
+# pattern was matched.
+zpty_read() {
+  local logvar=$1 pattern=$2
+  zpty -r -m zsh $logvar $pattern
+  local -i result=$?
+  zpty_handle_dputs $logvar
+  return result
+}
+
 zpty_flush() {
   local junk
   if zpty -r -t zsh junk \*; then
+    zpty_handle_dputs junk
     (( ZTST_verbose > 2 )) && print -n -u $ZTST_fd "$*: ${(V)junk}"
     while zpty -r -t zsh junk \* ; do
+      zpty_handle_dputs junk
       (( ZTST_verbose > 2 )) && print -n -u $ZTST_fd "${(V)junk}"
     done
     (( ZTST_verbose > 2 )) && print -u $ZTST_fd ''
   fi
 }
 
+# Detects, prints as "DPUTS:{<message>}\n", and removes any DPUTS
+# messages found in the parameter named $1.
+zpty_handle_dputs() {
+  local -nu log=$1
+  local -i lastend=0
+  while (( ${(N)${log[lastend+1,-1]}#*(#b)((<->:|) *.c:<->: *[$'\r\n'])} )); do
+    print -r "DPUTS:{${match[1][1,-2]}}"
+    log[lastend+$mbegin[1],lastend+$mend[1]]=""
+    [[ log[lastend+$mbegin[1]] != $'\n' ]] || log[lastend+$mbegin[1]]=""
+    lastend+=$mbegin[1]
+  done
+}
+
 zpty_run() {
   zpty -w zsh "$*"
-  zpty -r -m zsh log "*<PROMPT>*" || {
+  zpty_read log "*<PROMPT>*" || {
     print "prompt hasn't appeared."
     return 1
   }
@@ -130,7 +157,7 @@ comptesteval () {
 
      # zpty_flush Before comptesteval
      zpty -w zsh ". ${(q)tmp}"
-     zpty -r -m zsh log_eval "*<PROMPT>*" || {
+     zpty_read log_eval "*<PROMPT>*" || {
        print "prompt hasn't appeared."
        return 1
      }
@@ -144,7 +171,7 @@ comptesteval () {
 comptest () {
   input="$*"
   zpty -n -w zsh "$input"$'\C-Z'
-  zpty -r -m zsh log "*<WIDGET><finish>*<PROMPT>*" || {
+  zpty_read log "*<WIDGET><finish>*<PROMPT>*" || {
     print "failed to invoke finish widget."
     return 1
   }
@@ -184,7 +211,7 @@ zletest () {
     zpty -n -w zsh "$input"
   done
   zpty -n -w zsh $'\C-X'
-  zpty -r -m zsh log "*<WIDGET><finish>*<PROMPT>*" || {
+  zpty_read log "*<WIDGET><finish>*<PROMPT>*" || {
     print "failed to invoke finish widget."
     return 1
   }

diff --git a/Src/Zle/complete.c b/Src/Zle/complete.c
index 4bc048776..8a18e16e6 100644
--- a/Src/Zle/complete.c
+++ b/Src/Zle/complete.c
@@ -1339,6 +1339,7 @@ static const struct gsu_hash compstate_gsu =
 void
 makecompparams(void)
 {
+    DPUTS(1, "!!! makecompparams !!!");
     Param cpm;
     HashTable tht;
 
diff --git a/Src/Zle/zle_main.c b/Src/Zle/zle_main.c
index 37d587d02..6a5fabbbb 100644
--- a/Src/Zle/zle_main.c
+++ b/Src/Zle/zle_main.c
@@ -2123,6 +2123,7 @@ zleaftertrap(UNUSED(Hookdef dummy), UNUSED(void *dat))
 static char *
 zle_main_entry(int cmd, va_list ap)
 {
+    // DPUTS(1, "!!! zle_main_entry !!!");
     switch (cmd) {
     case ZLE_CMD_GET_LINE:
     {
diff --git a/Test/comptest b/Test/comptest
index c006c3efc..6bd5dd94f 100644
--- a/Test/comptest
+++ b/Test/comptest
@@ -127,6 +127,7 @@ zpty_flush() {
 # Detects, prints as "DPUTS:{<message>}\n", and removes any DPUTS
 # messages found in the parameter named $1.
 zpty_handle_dputs() {
+  return
   local -nu log=$1
   local -i lastend=0
   while (( ${(N)${log[lastend+1,-1]}#*(#b)((<->:|) *.c:<->: *[$'\r\n'])} )); do
@@ -183,7 +184,8 @@ comptest () {
     if [[ "$log" = (#b)*$'<LBUFFER>'(*)$'</LBUFFER>\r\n<RBUFFER>'(*)$'</RBUFFER>'* ]]; then
       print -lr "line: {$match[1]}{$match[2]}"
     fi
-    while (( ${(N)log#*(#b)(<LC><(??)><RC>(*)<EC>|<DESCRIPTION>(*)</DESCRIPTION>|<MESSAGE>(*)</MESSAGE>|<COMPADD>(*)</COMPADD>|<INSERT_POSITIONS>(*)</INSERT_POSITIONS>)} )); do
+    # while (( ${(N)log#*(#b)(<LC><(??)><RC>(*)<EC>|<DESCRIPTION>(*)</DESCRIPTION>|<MESSAGE>(*)</MESSAGE>|<COMPADD>(*)</COMPADD>|<INSERT_POSITIONS>(*)</INSERT_POSITIONS>)} )); do
+    while (( ${(N)log#*(#b)(<LC><(??)><RC>(*)<EC>|<DESCRIPTION>(*)</DESCRIPTION>|<MESSAGE>(*)</MESSAGE>|<COMPADD>(*)</COMPADD>|<INSERT_POSITIONS>(*)</INSERT_POSITIONS>|((<->:|) *.c:<->: *[$'\r\n']))} )); do
       log="${log[$mend[1]+1,-1]}"
       if (( 0 <= $mbegin[2] )); then
 	if [[ $match[2] != TC && $match[3] != \ # ]]; then
@@ -197,6 +199,8 @@ comptest () {
         print -lr "COMPADD:{${${match[6]}//[$'\r\n']/}}"
       elif (( 0 <= $mbegin[7] )); then
         print -lr "INSERT_POSITIONS:{${${match[7]}//[$'\r\n']/}}"
+      elif (( 0 <= $mbegin[8] )); then
+        print -lr "DPUTS:{${${match[8]}//[$'\r\n']/}}"
       fi
     done
   done