Re: serverizing a fat process with named pipes

[Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>]

Alexy Khrabrov wrote:
> I have a heavy process, an English parser loading megabytes of models,
> and then reading stdin, sentence per line, outputting the parsed text
> to the stdout.  How do I properly serverize it -- running in the
> background with <p1 >p2, those created with mkfifo p1 p2?  Looks like
> unless I wrap that in a kind of a loop, the processes exit at once.
> First I did this simulation:
> 
> mkfifo p1 p1
> cat <p1 >p2 &
> cat p2 &
> echo hi > p1
> 
> This does print "hi" and then both backgrounds exit.  If I wrap them
> in while true; do ... done & (& from the original goes outside), the
> simple one looks running continuously, but the parser takes a while
> starting up every time, apparently...

I think your problem's fairly straightforward:  the parser just needs to
open the file, read from it, and when it gets EOF on the input, close it
again.  It can then jump back to trying to open the file again.  It's
not really a shell problem, it's internal to the parser program.
Schematically (not complete code)...

for (;;) {
   FILE *file = fopen(p1);
   while (fread(file, ....) > 0) {
      /* parse and output */
   }
   fclose(file);
}

The programme will block at the fopen() until somebody writes to the
other side of the pipe.

You can do this sort of thing within the shell, if you need...  there's a
trick with a variable to get a file descriptor (this *is* working code
this time):

# initialize here
while true; do
   exec {fd}<p1  # script blocks here until someone writes to p1
   # something more exciting goes here...
   cat <&$fd
   exec {fd}<&- 
   print Restarting...
done

That's not quite the same as the loop you originally tried because
you've now got control of the file descriptor at the point where
the file opens and you hang on to it until you explicitly close it,
instead of relying on a process in the middle doing that for you.

If you're using a pipe on the output end you'll need to watch out for
the server process getting SIGPIPE if the reading process exits.  You
can simply ignore that ("trap '' PIPE" in the shell) and rely on a write
failing to terminate the current iteration and go back and wait to be
able to open the output again (but you do either need to handle the
SIGPIPE or the failing write).

-- 
Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/