Re: [PATCH] tests: improve readability

[Philippe Altherr <philippe.altherr@xxxxxxxxx>]

mikael and i were talking about how the test output is not as readable
as we'd like.

I came to the same conclusion and developed the following sed command to improve the test output readability.

     sed -E \

        -e "s/^([.]\/.*[.]ztst: skipped .*)/\x1b[1m\x1b[36m&\x1b[0m/" \
        -e "s/^([.]\/.*[.]ztst: .*)/\x1b[1m&\x1b[0m/" \
        -e "s/^(Was testing: )(.*)/\x1b[1m\1\x1b[31m\2\x1b[0m/" \
        -e "s/^(Test [.]\/.*[.]ztst failed: )(.*)/\x1b[1m\1\x1b[31m\2\x1b[0m/" \
        -e "s/^([-][^-].*)/\x1b[31m&\x1b[0m/" \
        -e "s/^([+][^+].*)/\x1b[32m&\x1b[0m/" \
        -e "s/^(Test case skipped: .*)/\x1b[36m&\x1b[0m/" \
        -e "s/^([*]+)/\x1b[1m&\x1b[0m/" \
        -e "s/^([0-9]+ successful test scripts?, 0 failures, 0 skipped)/\x1b[1m\x1b[32m&\x1b[0m/" \
        -e "s/^([0-9]+ successful test scripts?, 0 failures), (.+ skipped)/\x1b[1m\x1b[32m\1\x1b[30m, \x1b[36m\2\x1b[0m/" \
        -e "s/^([0-9]+ successful test scripts?), (.+ failures?), (0 skipped)/\x1b[1m\1, \x1b[31m\2\x1b[30m, \3\x1b[0m/" \
        -e "s/^([0-9]+ successful test scripts?), (.+ failures?), (.+ skipped)/\x1b[1m\1, \x1b[31m\2\x1b[30m, \x1b[36m\3\x1b[0m/"

One important difference with your patch is that it puts more weight on highlighting failed test cases within test files than just highlinging the failed test files. This is especially important for test files run with ZTST_continue=1 (which I usually always do) because there are often multiple test cases that fail and it's important to be able to quickly identify them.

Here is a side-by-side comparison of the output for A03quoting.ztst after adding a bogus "X" in the expected output of two test cases and running the test with ZTST_continue=1.

Your patch

My sed command

A03quoting.ztst: starting. --- expected +++ actual @@ -1,7 +1,7 @@  16#42 -16#53X +16#53  16#5C -16#4DX +16#4D  16#42  16#53  16#DC Test A03quoting.ztst failed: output differs from expected as shown above for:   chars=$(print -r $'BS\\MBS\M-\\')   for (( i = 1; i <= $#chars; i++ )); do     char=$chars[$i]     print $(( [#16] #char ))   done Was testing: $'-style quote with metafied backslash A03quoting.ztst: test failed. --- expected +++ actual @@ -1,6 +1,6 @@  :  16#61 -16#62X +16#62  16#0  16#63  16#64 Test A03quoting.ztst failed: output differs from expected as shown above for:   null1="$(print -r a$'b\0c'd)"   null2="$(setopt posixstrings; print -r a$'b\0c'd)"   for string in $null1 $null2; do     print ":"     for (( i = 1; i <= $#string; i++ )); do       char=$string[$i]       print $(( [#16] #char ))     done   done Was testing: Embedded null characters in $'...' strings. A03quoting.ztst: test failed.

./A03quoting.ztst: starting. --- /tmp/zsh.ztst.74957/ztst.out 2026-06-05 12:43:51 +++ /tmp/zsh.ztst.74957/ztst.tout 2026-06-05 12:43:51 @@ -1,7 +1,7 @@  16#42 -16#53X +16#53  16#5C -16#4DX +16#4D  16#42  16#53  16#DC Test ./A03quoting.ztst failed: output differs from expected as shown above for:   chars=$(print -r $'BS\\MBS\M-\\')   for (( i = 1; i <= $#chars; i++ )); do     char=$chars[$i]     print $(( [#16] #char ))   done Was testing: $'-style quote with metafied backslash ./A03quoting.ztst: test failed. --- /tmp/zsh.ztst.74957/ztst.out 2026-06-05 12:43:51 +++ /tmp/zsh.ztst.74957/ztst.tout 2026-06-05 12:43:51 @@ -1,6 +1,6 @@  :  16#61 -16#62X +16#62  16#0  16#63  16#64 Test ./A03quoting.ztst failed: output differs from expected as shown above for:   null1="$(print -r a$'b\0c'd)"   null2="$(setopt posixstrings; print -r a$'b\0c'd)"   for string in $null1 $null2; do     print ":"     for (( i = 1; i <= $#string; i++ )); do       char=$string[$i]       print $(( [#16] #char ))     done   done Was testing: Embedded null characters in $'...' strings. ./A03quoting.ztst: test failed.

Your patch only puts "A03quoting.ztst: test failed." in red. My sed command instead puts in red and bold, the names of the failing test cases and the reasons for their failures.

For each test file, the output starts with "<filename>: starting." and ends with "<filename>: all tests successful.", if all test cases were successful, and with "<filename>: test failed." otherwise. My sed command puts these messages in bold (but doesn't color them). This is useful to identify the start and stop of each test file and distinguish these messages from other output (some tests are noisy even in case of success). With the addition of blank lines after each test file, this boldening becomes less important but I think it would still be useful.

The message "<filename>: test failed." is added after each failed test case. In my opinion, an improvement would be to output it only once at the end of the test file. It could include the number of failed test cases and maybe list their names, like you now list the names of the failed test files at the very end.

Below are a few more side-by-side comparisons of when

• there are failures and skipped test cases and files
• there are failures but no skipped files
• there are no failures but skipped files
• all tests were successful.

Your patch

My sed command (with manually added blank lines)

[…] V07pcre.ztst: starting. V07pcre.ztst: skipped (the zsh/pcre module was disabled by configure (see config.modules)) V08zpty.ztst: starting. V08zpty.ztst: all tests successful. V09datetime.ztst: starting. Test case skipped: strftime extensions not supported Test case skipped: strftime extensions not supported V09datetime.ztst: all tests successful. […] Z04zgetopt.ztst: starting. Z04zgetopt.ztst: all tests successful. ************************************************ 77 successful test scripts, 1 failure, 2 skipped failing test scripts:   A03quoting.ztst ************************************************

[…] ./V07pcre.ztst: starting. ./V07pcre.ztst: skipped (the zsh/pcre module was disabled by configure (see config.modules)) ./V08zpty.ztst: starting. ./V08zpty.ztst: all tests successful. ./V09datetime.ztst: starting. Test case skipped: strftime extensions not supported Test case skipped: strftime extensions not supported ./V09datetime.ztst: all tests successful. […] ./Z04zgetopt.ztst: starting. ./Z04zgetopt.ztst: all tests successful. ************************************** 72 successful test scripts, 1 failure, 2 skipped **************************************

Your patch	My sed command (with manually added blank lines)
[…] *********************************************** 8 successful test scripts, 1 failure, 0 skipped failing test scripts:   A03quoting.ztst ***********************************************	[…] ************************************** 8 successful test scripts, 1 failure, 0 skipped **************************************

Your patch	My sed command (with manually added blank lines)
[…] ************************************************* 13 successful test scripts, 0 failures, 1 skipped *************************************************	[…] ************************************** 13 successful test scripts, 0 failures, 1 skipped **************************************

Your patch	My sed command (with manually added blank lines)
[…] ************************************************* 15 successful test scripts, 0 failures, 0 skipped *************************************************	[…] ************************************** 13 successful test scripts, 0 failures, 0 skipped **************************************

My recommendations/wishes:

• Put at least the main summary line and the star lines in bold
• Use more fine grained coloring for the summary line (see results of my sed command)
• Put the star lines in red if there are any failures and otherwise in green (and maybe in cyan if there are only skipped files)

Would it be useful to also list (in cyan) the skipped files?

Philippe