PATCH: 3.1.6-pws-10: Initial testing stuff

[Peter Stephenson <pws@xxxxxxxxxxxxxxxxxxxxxxxx>]

This is my first go at providing some test support.  It is very inchoate:
the only test file is Test/cd.ztst, which is supposed to act as both
documentation and a model for others; it is not supposed to complete in
itself, even.  The test driver script is Test/ztst.zsh; run `make tests' in
Test to run the test(s).  Note that it uses the zsh you have just built in
Src (this avoids all the name transformation problems), but you will have
to have installed any required modules --- parameter is required for ease
of option handling.

Comments on the basic form of this are welcome.  If it seems OK (it
obviously needs more work as well as more tests), suggestions for tests to
perform, as well as the tests themselves, will be welcome.

At some point it will need to be enhanced to test job control and zle,
since those are two areas needing a lot of testing.  It may be possible to
do this with expect, which is reasonably widely available, or some
perl module with equivalent functionality, or we could even add our own
perl module.

(I now finally have my own CVS archive.  This will no doubt be another
source of problems.)

Index: configure.in
===================================================================
RCS file: /home/pws/CVSROOT/projects/zsh/configure.in,v
retrieving revision 1.7
diff -u -r1.7 configure.in
--- configure.in	1999/12/06 23:58:38	1.7
+++ configure.in	1999/12/07 21:37:50
@@ -1555,7 +1555,7 @@
 no_create=yes
 
 AC_OUTPUT(Config/defs.mk Makefile Doc/Makefile Etc/Makefile Src/Makefile \
-Completion/Makefile Functions/Makefile, \
+Test/Makefile Completion/Makefile Functions/Makefile, \
 [test -z "$CONFIG_HEADERS" || echo > stamp-h])
 
 dnl The standard config.status is missing some essential features.
Index: Etc/zsh-development-guide
===================================================================
RCS file: /home/pws/CVSROOT/projects/zsh/Etc/zsh-development-guide,v
retrieving revision 1.2
diff -u -r1.2 zsh-development-guide
--- Etc/zsh-development-guide	1999/12/03 19:12:10	1.2
+++ Etc/zsh-development-guide	1999/12/07 21:58:24
@@ -40,6 +40,30 @@
   archive, send them to the mailing list with a Subject: line starting
   with "PATCH:".
 
+Testing
+-------
+
+* Because zsh has a huge number of different options and interacts with
+  a wide range of human and artificial life, it is very difficult to
+  test the shell thoroughly.  For this purpose, the Test subdirectory
+  exists.  It consists of a driver script (ztst.zsh) and various test
+  files (*.ztst) in a format which is described in cd.ztst, which acts
+  as a template.  It is designed to make it easy to provide input to
+  chunks of shell code and to test the corresponding standard output,
+  error output and exit status.
+
+* There is not much there yet, but please don't let that put you off adding
+  tests for basic syntactic features, builtins, options etc. which you
+  know to be flakey or to have had difficulties in the past.  Better
+  support for testing job control and interactive features is expected
+  to follow eventually (this may require additional external software
+  e.g. `expect').
+
+* The directory is not part of the usual process of building and
+  installation.  To run the tests, go to Test and `make test'.  Please
+  report any errors with all the usual information about the zsh version
+  and the system you are using.
+
 C coding style
 --------------
 
Index: Test/Makefile.in
===================================================================
RCS file: Makefile.in
diff -N Makefile.in
--- /dev/null	Tue May  5 21:32:27 1998
+++ Test/Makefile.in	Tue Dec  7 21:52:55 1999
@@ -0,0 +1,62 @@
+#
+# Makefile for Test subdirectory
+#
+# Copyright (c) 1999 Peter Stephensons
+# All rights reserved.
+#
+# Permission is hereby granted, without written agreement and without
+# license or royalty fees, to use, copy, modify, and distribute this
+# software and to distribute modified versions of this software for any
+# purpose, provided that the above copyright notice and the following
+# two paragraphs appear in all copies of this software.
+#
+# In no event shall Peter Stephenson or the Zsh Development Group be liable
+# to any party for direct, indirect, special, incidental, or consequential
+# damages arising out of the use of this software and its documentation,
+# even if Peter Stephenson and the Zsh Development Group have been advised of
+# the possibility of such damage.
+#
+# Peter Stephenson and the Zsh Development Group specifically disclaim any
+# warranties, including, but not limited to, the implied warranties of
+# merchantability and fitness for a particular purpose.  The software
+# provided hereunder is on an "as is" basis, and Peter Stephenson and the
+# Zsh Development Group have no obligation to provide maintenance,
+# support, updates, enhancements, or modifications.
+#
+
+subdir = Test
+dir_top = ..
+SUBDIRS = 
+
+@VERSION_MK@
+
+# source/build directories
+VPATH           = @srcdir@
+sdir            = @srcdir@
+sdir_top        = @top_srcdir@
+INSTALL         = @INSTALL@
+
+@DEFS_MK@
+
+# ========== DEPENDENCIES FOR TESTING ==========
+
+tests:
+	for f in *.ztst; do \
+	  ../Src/zsh ztst.zsh $$f; \
+	done
+
+# ========== DEPENDENCIES FOR CLEANUP ==========
+
+@CLEAN_MK@
+
+mostlyclean-here:
+	rm -rf *.tmp
+
+distclean-here:
+	rm -f Makefile
+
+realclean-here:
+
+# ========== DEPENDENCIES FOR MAINTENANCE ==========
+
+@CONFIG_MK@
Index: Test/cd.ztst
===================================================================
RCS file: cd.ztst
diff -N cd.ztst
--- /dev/null	Tue May  5 21:32:27 1998
+++ Test/cd.ztst	Tue Dec  7 21:56:53 1999
@@ -0,0 +1,97 @@
+# This file serves as a model for how to write tests, so is more heavily
+# commented that the others.  All tests are run in the Test subdirectory
+# of the distribution, which must be writable.  They should end with
+# the suffix `.ztst': this is not required by the test harness itself,
+# but it is needed by the Makefile to run all the tests.
+
+# Blank lines with no other special meaning (e.g. separating chunks of
+# code) and all those with a `#' in the first column are ignored.
+
+# All section names start with a % in the first column.  The names
+# must be in the expected order, though not all sections are required.
+# The sections are %prep (preparatory setup:  code executed should return
+# status 0, but no other tests are performed), %test (the main tests), and
+# %clean (to cleanup: the code is simply unconditionally executed).
+#
+# Literal shell code to be evaluated must be indented with any number
+# of spaces and/or tabs, to differentiate it from tags with a special
+# meaning to the test harness.  Note that this is true even in sections
+# where there are no such tags.  Also note that file descriptor 9
+# is reserved for input from the test script; if ZTST_verbose is set,
+# output is sent to the original stdout via fd 8.  Option settings
+# are preserved between the execution of different code chunks;
+# initially, all standard zsh options (the effect of `emulate -R zsh')
+# are set.
+
+%prep
+# This optional section prepares the test, creating directories and files
+# and so on.  Chunks of code are separated by blank lines (which is not
+# necessary before the end of the section); each chunk of code is evaluated
+# in one go and must return status 0, or the preparation is deemed to have
+# failed and the test ends with an appropriate error message.  Standard
+# output from this section is redirected to /dev/null, but standard error
+# is not redirected.
+#
+# Tests should use subdirectories ending in `.tmp'.  These will be
+# removed with all the contents even if the test is aborted.
+ mkdir cdtst.tmp cdtst.tmp/real cdtst.tmp/sub
+
+ ln -s ../real cdtst.tmp/sub/fake
+
+ mydir=$PWD
+
+%test
+# This is where the tests are run.  It consists of blocks separated
+# by blank lines.  Each block has the same format and there may be any
+# number of them.  It consists of indented code, plus optional sets of lines
+# beginning '<', '>' and '?' which may appear in any order.  These correspond
+# to stdin (fed to the code), stdout (compared with code output) and
+# stderr (compared with code error output) respectively.  These subblocks
+# may occur in any order, but the natural one is: code, stdin, stdout,
+# stderr.
+#
+# The rules for '<', '>' and '?' lines are the same: only the first
+# character is stripped, with subsequent whitespace being significant;
+# lines are subject to ordinary quoted shell expansion (i.e. not globbing).
+#
+# Each chunk of indented code is to be evaluated in one go and is to
+# be followed by a line starting (in the first column) with
+# the expected status returned by the code when run, or - if it is
+# irrelevant.  This can be followed by a `:' and a message describing the
+# test, which will be printed if the test fails, along with a
+# description of the failure that occurred.  The `:' and message are
+# optional, but highly recommended.
+#
+# If either or both of the '>' and '?' sets of lines is absent, it is
+# assumed the corresponding output should be empty and it is an error if it
+# is not.  If '<' is empty, stdin is an empty (but opened) file.
+#
+# TODO: flags to the post-code status line indicating that diffs are
+# not to be performed.
+ cd cdtst.tmp/sub/fake &&
+ pwd &&
+ print $PWD
+0:Preserving symbolic links in the current directory string
+>$mydir/cdtst.tmp/sub/fake
+>$mydir/cdtst.tmp/sub/fake
+
+ cd ../../.. &&
+ pwd &&
+ print $PWD
+0:Changing directory up through symbolic links without following them
+>$mydir
+>$mydir
+
+ setopt chaselinks
+ cd cdtst.tmp/sub/fake &&
+ pwd &&
+ print $PWD
+0:Resolving symbolic links with chaselinks set
+>$mydir/cdtst.tmp/real
+>$mydir/cdtst.tmp/real
+
+%clean
+# This optional section cleans up after the test, if necessary,
+# e.g. killing processes etc.  This is in addition to the removal of *.tmp
+# subdirectories.  This is essentially like %prep, except that status
+# return values are ignored.
Index: Test/ztst.zsh
===================================================================
RCS file: ztst.zsh
diff -N ztst.zsh
--- /dev/null	Tue May  5 21:32:27 1998
+++ Test/ztst.zsh	Tue Dec  7 21:45:35 1999
@@ -0,0 +1,316 @@
+#!/usr/local/bin/zsh -f
+# The line above is just for convenience.  Normally tests will be run using
+# a specified version of zsh.  With dynamic loading, any required libraries
+# must already have been installed in that case.
+#
+# Takes one argument: the name of the test file.  Currently only one such
+# file will be processed each time ztst.zsh is run.  This is slower, but
+# much safer in terms of preserving the correct status.
+# To avoid namespace pollution, all functions and parameters used
+# only by the script begin with ZTST_.
+#
+# Options (without arguments) may precede the test file argument; these
+# are interpreted as shell options to set.  -x is probably the most useful.
+
+# Produce verbose messages if non-zero.
+# If 1, produce reports of tests executed; if 2, also report on progress.
+ZTST_verbose=0
+
+# We require all options to be reset, not just emulation options.
+# Unfortunately, due to the crud which may be in /etc/zshenv this might
+# still not be good enough.  Maybe we should trick it somehow.
+emulate -R zsh
+
+# We need to be able to save and restore the options used in the test.
+# We use the $options variable of the parameter module for this.
+zmodload -i parameter
+
+# Note that both the following are regular arrays, since we only use them
+# in whole array assignments to/from $options.
+# Options set in test code (i.e. by default all standard options)
+ZTST_testopts=(${(kv)options})
+
+setopt extendedglob nonomatch
+while [[ $1 = [-+]* ]]; do
+  set $1
+  shift
+done
+# Options set in main script
+ZTST_mainopts=(${(kv)options})
+
+# We run in the current directory, so remember it.
+ZTST_testdir=$PWD
+ZTST_testname=$1
+
+# Temporary files for redirection inside tests.
+ZTST_in=${TMPPREFIX-:/tmp/zsh}.ztst.in.$$
+# hold the expected output
+ZTST_out=${TMPPREFIX-:/tmp/zsh}.ztst.out.$$
+ZTST_err=${TMPPREFIX-:/tmp/zsh}.ztst.err.$$
+# hold the actual output from the test
+ZTST_tout=${TMPPREFIX-:/tmp/zsh}.ztst.tout.$$
+ZTST_terr=${TMPPREFIX-:/tmp/zsh}.ztst.terr.$$
+
+ZTST_cleanup() {
+  rm -rf $ZTST_testdir/dummy.tmp $ZTST_testdir/*.tmp \
+         $ZTST_in $ZTST_out $ZTST_err $ZTST_tout $ZTST_terr
+}
+
+# This cleanup always gets performed, even if we abort.  Later,
+# we should try and arrange that any test-specific cleanup
+# always gets called as well.
+trap - 'print cleaning up...
+ZTST_cleanup' INT QUIT TERM
+# Make sure it's clean now.
+rm -rf dummy.tmp *.tmp
+
+# Report failure.  Note that all output regarding the tests goes to stdout.
+# That saves an unpleasant mixture of stdout and stderr to sort out.
+ZTST_testfailed() {
+  print "Test $ZTST_testname failed: $1"
+  if [[ -n $ZTST_message ]]; then
+    print "Was testing: $ZTST_message"
+  fi
+  ZTST_cleanup
+  exit 1
+}
+
+# Print messages if $ZTST_verbose is non-empty
+ZTST_verbose() {
+  local lev=$1
+  shift
+  [[ -n $ZTST_verbose && $ZTST_verbose -ge $lev ]] && print $* >&8
+}
+
+[[ ! -r $ZTST_testname ]] && ZTST_testfailed "can't read test file."
+
+[[ -n $ZTST_verbose && $ZTST_verbose -ge 0 ]] && exec 8>&1
+exec 9<$ZTST_testname
+
+# The current line read from the test file.
+ZTST_curline=''
+# The current section being run
+ZTST_cursect=''
+
+# Get a new input line.  Don't mangle spaces; set IFS locally to empty.
+# We shall skip comments at this level.
+ZTST_getline() {
+  local IFS=
+  while true; do
+    read ZTST_curline <&9 || return 1
+    [[ $ZTST_curline == \#* ]] || return 0
+  done
+}
+
+# Get the name of the section.  It may already have been read into
+# $curline, or we may have to skip some initial comments to find it.
+ZTST_getsect() {
+  local match mbegin mend
+
+  while [[ $ZTST_curline != '%'(#b)([[:alnum:]]##)* ]]; do
+    ZTST_getline || return 1
+    [[ $ZTST_curline = [[:blank:]]# ]] && continue
+    if [[ $ZTST_curline != '%'[[:alnum:]]##* ]]; then
+      ZTST_testfailed "bad line found before or after section:
+$ZTST_curline"
+    fi
+  done
+  # have the next line ready waiting
+  ZTST_getline
+  ZTST_cursect=${match[1]}
+  ZTST_verbose 2 "ZTST_getsect: read section name: $ZTST_cursect"
+  return 0
+}
+
+# Read in an indented code chunk for execution
+ZTST_getchunk() {
+  # Code chunks are always separated by blank lines or the
+  # end of a section, so if we already have a piece of code,
+  # we keep it.  Currently that shouldn't actually happen.
+  ZTST_code=''
+  # First find the chunk.
+  while [[ $ZTST_curline = [[:blank:]]# ]]; do
+    ZTST_getline || break
+  done
+  while [[ $ZTST_curline = [[:blank:]]##[^[:blank:]]* ]]; do
+    ZTST_code="${ZTST_code:+${ZTST_code}
+}${ZTST_curline}"
+    ZTST_getline || break
+  done
+  ZTST_verbose 2 "ZTST_getchunk: read code chunk:
+$ZTST_code"
+  [[ -n $ZTST_code ]]
+}
+
+# Read in a piece for redirection.
+ZTST_getredir() {
+  local char=${ZTST_curline[1]}
+  ZTST_redir=${ZTST_curline[2,-1]}
+  while ZTST_getline; do
+    [[ $ZTST_curline[1] = $char ]] || break
+    ZTST_redir="${ZTST_redir}
+${ZTST_curline[2,-1]}"
+  done
+  ZTST_verbose 2 "ZTST_getredir: read redir for '$char':
+$ZTST_redir"
+}
+
+# Execute an indented chunk.  Redirections will already have
+# been set up, but we need to handle the options.
+ZTST_execchunk() {
+  options=($ZTST_testopts)
+  eval "$ZTST_code"
+  ZTST_status=$?
+  ZTST_verbose 2 "ZTST_execchunk: status $ZTST_status"
+  ZTST_testopts=(${(kv)options})
+  options=($ZTST_mainopts)
+  return $ZTST_status
+}
+
+# Functions for preparation and cleaning.
+# When cleaning up (non-zero string argument), we ignore status.
+ZTST_prepclean() {
+  # Execute indented code chunks.
+  while ZTST_getchunk; do
+    ZTST_execchunk >/dev/null || [[ -n $1 ]] ||
+    ZTST_testfailed "non-zero status from preparation code:
+$ZTST_code"
+  done
+}
+
+ZTST_test() {
+  local last match mbegin mend found
+
+  while true; do
+    rm -f $ZTST_in $ZTST_out $ZTST_err
+    touch $ZTST_in $ZTST_out $ZTST_err
+    ZTST_message=''
+    found=0
+
+    ZTST_verbose 2 "ZTST_test: looking for new test"
+
+    while true; do
+      ZTST_verbose 2 "ZTST_test: examining line:
+$ZTST_curline"
+      case $ZTST_curline in
+	%*) if [[ $found = 0 ]]; then
+	      break 2
+	    else
+	      last=1
+	      break
+	    fi
+	    ;;
+	[[:space:]]#)
+	    if [[ $found = 0 ]]; then
+	      ZTST_getline || break 2
+	      continue
+	    else
+	      break
+	    fi
+	    ;;
+	[[:space:]]##[^[:space:]]*) ZTST_getchunk
+	  [[ $ZTST_curline != [-0-9]* ]] &&
+	  ZTST_testfailed "expecting test status at:
+$ZTST_curline"
+          ZTST_xstatus=$ZTST_curline
+	  if [[ $ZTST_curline == (#b)([^:]##):(*) ]]; then
+	    ZTST_xstatus=$match[1]
+	    ZTST_message=$match[2]
+	  fi
+	  ZTST_getline
+	  found=1
+	  ;;
+	'<'*) ZTST_getredir
+	  print -r "${(e)ZTST_redir}" >>$ZTST_in
+	  found=1
+	  ;;
+	'>'*) ZTST_getredir
+          print -r "${(e)ZTST_redir}" >>$ZTST_out
+	  found=1
+	  ;;
+	'?'*) ZTST_getredir
+	  print -r "${(e)ZTST_redir}" >>$ZTST_err
+	  found=1
+	  ;;
+	*) ZTST_testfailed "bad line in test block:
+$ZTST_curline"
+          ;;
+      esac
+    done
+
+    # If we found some code to execute...
+    if [[ -n $ZTST_code ]]; then
+      ZTST_verbose 1 "Running test:
+$ZTST_message"
+      ZTST_verbose 2 "ZTST_test: expecting status: $ZTST_xstatus"
+
+      ZTST_execchunk <$ZTST_in >$ZTST_tout 2>$ZTST_terr
+
+      # First check we got the right status, if specified.
+      if [[ $ZTST_xstatus != - && $ZTST_xstatus != $ZTST_status ]]; then
+	ZTST_testfailed "bad status $ZTST_status, expected $ZTST_xstatus from:
+$ZTST_code"
+      fi
+
+      ZTST_verbose 2 "ZTST_test: test produced standard output:
+$(<$ZTST_tout)
+ZTST_test: and standard error:
+$(<$ZTST_terr)"
+
+      # Now check output and error.
+      if ! diff -c $ZTST_out $ZTST_tout; then
+	ZTST_testfailed "output differs from expected as shown above for:
+$ZTST_code"
+      fi
+      if ! diff -c $ZTST_err $ZTST_terr; then
+	ZTST_testfailed "error output differs from expected as shown above for:
+$ZTST_code"
+      fi
+    fi
+    ZTST_verbose 1 "Test successful."
+    [[ -n $last ]] && break
+  done
+
+  ZTST_verbose 2 "ZTST_test: all tests successful"
+
+  # reset message to keep ZTST_testfailed output correct
+  ZTST_message=''
+}
+
+
+# Remember which sections we've done.
+typeset -A ZTST_sects
+ZTST_sects=(prep 0 test 0 clean 0)
+
+# Now go through all the different sections until the end.
+while ZTST_getsect; do
+  case $ZTST_cursect in
+    prep) if (( ${ZTST_sects[prep]} + ${ZTST_sects[test]} + \
+	        ${ZTST_sects[clean]} )); then
+	    ZTST_testfailed "\`prep' section must come first"
+	  fi
+	  ZTST_prepclean
+	  ZTST_sects[prep]=1
+	  ;;
+    test)
+	  if (( ${ZTST_sects[test]} + ${ZTST_sects[clean]} )); then
+	    ZTST_testfailed "bad placement of \`test' section"
+	  fi
+	  ZTST_test
+	  ZTST_sects[test]=1
+	  ;;
+    clean)
+	   if (( ${ZTST_sects[test]} == 0 || ${ZTST_sects[clean]} )); then
+	     ZTST_testfailed "bad use of \`clean' section"
+	   fi
+	   ZTST_prepclean 1
+	   ZTST_sects[clean]=1
+	   ;;
+    *) ZTST_testfailed "bad section name: $ZTST_cursect"
+       ;;
+  esac
+done
+
+print "$ZTST_testname: all tests successful."
+ZTST_cleanup
+exit 0

-- 
Peter Stephenson <pws@xxxxxxxxxxxxxxxxxxxxxxxx>