Re: Writing Interpreter Testing Framework

[Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>]

"Johann 'Myrkraverk' Oskarsson" wrote:
> For the sake of argument, let's say I'm testing an interpreter* which
> basically means I want to run an application, feed it some commands
> from a file to STDIN, and depending on the results on its STDOUT, I
> want to feed some other file.
> 
> That is, something like the following interactive session:
> 
> % interpreter < file_of_commands_A
> Some output from interpreter: B
> % interpreter < file_of_commands_A < file_of_commands_B
> 
> Though I'm not excacly sure the above will work, and, if possible, I'd
> like to skip running the interpreter again, to preserve all possible
> random-set states.

I'm assuming you're interested in an interactive zsh way of doing this,
since you've asked the question here.  There are lots of script based
ways from expect to Perl libraries designed to do this which I will
ignore; but if you're really interested in robust scripts and don't care
about interactive use, that's probably a better way to go.  I'd
recommend Perl or another "real" language of your choice rather than
expect, since TCL is otherwise rather limited.  (Our test engineer has
just started writing scripts in Perl rather than expect and apparently
it all works much better.  I'm sure you can do it in Python, too.)

The standard zsh/ksh way of doing this is with coprocesses.  Here's a
trivial example with an "interpreter" that is just a simple subshell
loop that modifies the input and spits it out.  I've stripped
intermediate prompts for clarity:

% coproc (while read line; do 
print -r "You said: $line"
done)
% print this is a message >&p
% read input <&p
% print $input
You said: this is a message

If you need the "interpreter" to run in a (pseudo)terminal, investigate
the zpty command provided by the zsh/zpty module and documented in the
zshmodules manual page.  This works similarly to coprocesses, although
the syntax is different.

If you need the output to go straight to the (real) terminal, you need
to be a bit cleverer.  You can get some of the way by using some
intermediate stage for the input, for example a named pipe:

% mkfifo ~/tmp/fifo
% (while true; do;
while read line; do
print -r "You said: $line"
done <~/tmp/fifo     
done) &
% print Hello >~/tmp/fifo             
You said: Hello
% print Goodbye >~/tmp/fifo
You said: Goodbye

There are two subtleties here.  First, the double "while" loop.  That's
because the one-off prints to the FIFO cause an end-of-file, which
terminates the inner loop.  You'd need to take account of that in any
interpreter.

Second, the output from the background process is asynchronous and the
shell isn't expecting it, so although I showed the output as neat lines
between the shell intput you won't actually get that.

If you still want better interaction with the shell, you might want to
consider using the TCP framework supplied with the shell and documented
in the zshtcpsys manual entry.  It works most naturally if your
interpreter is running on a TCP port.  See below for a simple-minded
function that does this: it's not that hard to knock up a daemon to wrap
an interpreter that does that, though you'd better make sure you have a
good firewall if you go this route.  However, the system allows you to
come in at a lower level.  For example:

  # Make two FIFOs: we need that because FIFOs get confused if two
  # processes read from the same one.  That's one advantage of a real
  # TCP solution: you can use one file descriptor with no problem.
  % mkfifo ~/tmp/infifo
  % mkfifo ~/tmp/outfifo
  # Start the "interpreter".  Note we don't need any hacky extra loop this
  # time.
  % (while read line; do
  print -r "You said: $line"
  done <~/tmp/infifo >~/tmp/outfifo) &
  # Make fd 5 our output to this interpreter's input.
  % exec 5>~/tmp/infifo 
  # Open a session called "output" on this fd
  % tcp_open -f 5 output
  Session output (fd 5) opened OK.
  Setting default TCP session output
  # Make fd 4 our input from the interpreter's output.
  % exec 4<~/tmp/outfifo
  # Open a session called "input" on this fd
  % tcp_open -f 4 input
  Session input (fd 4) opened OK.

Now comes the good bit:

  % tcp_send Hello
  <-[input] You said: Hello

(You might see an empty line before the reply comes back if the line
editor starts up again before the reply is ready, but the display will
still be cleared for the output.)

tcp_send sends output to the default session which, as you can see from
the messages above, is the session "output" which writes to the
interpreter's input.  When we opened the session "input", the shell
automatically supplied the fd 4 to the line editor (using the builtin
"zle -F").  Now the shell listens for input not just on the editing
terminal, but on that fd.  When input comes, it alerts the TCP system,
which clears the terminal and displays the input with the prompt
"<-[input] " (this is configurable).

When you want to interact with the interpreter programmatically, you can
write functions like this:

fn() {
     emulate -L zsh
     setopt localoptions
     local -a match mbegin mend

     tcp_send "This is a message"
     if tcp_expect -s input '*You said: (#b)(*)'; then
          print -r "It said I said: $match[1]"
     fi
}

(Note that because the input and output are different sessions in our
FIFO approach you need to tell tcp_expect the session to use
explicitly.)  You'll see that you still get the raw line echoed; add
"local TCP_SILENT=1" to the top of the function to stop that.

If you are happy about the security aspect, here's how to use the full
majesty of TCP to do this:

# We need the interpreter to be a single command...
% interpreter() {
  while read line; do
    print -r "You said: $line"
  done
}
# Start the interpreter running on a random port
% tcp_proxy 5114 interpreter &
# We are now done with the interpreter... in fact, all the above
# could be in a different terminal or even on a different host.
# Now start the session "session" to talk to the interpreter
% tcp_open localhost 5114 session
Session session (host localhost, port 5114 fd 3) opened OK.
Setting default TCP session session
% tcp_send Hello
<-[session] You said: Hello

The tcp_expect stuff works as before, though this time you've only got
one session to worry about so don't need any "-s" arguments.

I use this system all the time at work for interacting with my company's
range of digital radio chips and I find it highly flexible.

-- 
Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/