Puzzling coproc descriptor problem

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

This got kind of long as I made notes on my experiments.  For the real
potentially fixable problem, skip down to after "END FIRST EXAMPLE",
but I'm also puzzled about #3 in that first example.

Fiddling with Nikolai Weibull's question about capturing both stdout and
stderr in separate variables ... the following ought to work but does
not for the reasons noted:

## BEGIN FIRST EXAMPLE SCRIPT

# First set up a coproc to capture stderr
coproc { err="$(cat)"; print -r ${(qq)err} }

# Problem #1 is the subshell spawned for { ... } is holding open the
# stdin/stdout descriptors, so $(cat) never sees EOF on stdin.  It
# works for a single line of data with "read -E" in place of cat.
# Hmm, pdksh 5.2 and ksh93 also seem to have this same problem, so it
# may be endemic to coprocesses, but let's finish the example.

# Copy and close descriptors in the parent
exec {e0}<&p {e1}>&p
coproc :

# Now set up a coproc to capture stdout.  Note we have to close the
# descriptors for the first coproc inside the scope of the second; this
# is expected and not a problem with the internals.
coproc { exec {e0}<&- {e1}>&-;
         out="$(cat)"; print -r ${(qq)out} }

# Problem #2 same as #1.

# Copy descriptors in the parent
exec {o0}<&p {o1}>&p

# Problem #3 is that at this point, according to the "lsof" command on
# linux, the descriptor identified by $o0 is open for *writing* when it
# should be open for *reading*.  $e0, which was created by exactly the
# same form of exec, is correctly open for reading.  $o1 and $e1 are
# both correctly open for writing.
#
## torch% print $o0 $e0
## 14 12
## torch% lsof -p $$ | egrep '(12|14)[rwu]'
## zsh     13449 schaefer   12r  FIFO    0,7          9045146 pipe
## zsh     13449 schaefer   14w  FIFO    0,7          9045462 pipe
#

# Close descriptors in the parent
coproc :

# Run the command with output to the appropriate coprocs ...
{ print THIS IS STDOUT; print -u2 THIS IS STDERR } >&$o1 2>&$e1

# ... and close those descriptors to close the input of $(cat),
# except see problem #1 so this is actually futile.
exec {o1}>&- {e1}>&-

# Finally, read from the two coproc outputs (this hangs forever)
out="$(cat <&$o0)"
err="$(cat <&$e0)"

# And close the read descriptors at the end, not strictly necessary
exec {o0}<&- {e0}<&-

## END FIRST EXAMPLE SCRIPT


OK, so using a process substitution to read from the coproc input is
not going to work, even in other shells.  I should be able to get
around that by using a loop on "read" that stays all in the current
shell.  Unfortunately, that doesn't work either:

## BEGIN SECOND EXAMPLE SCRIPT

coproc { while { IFS= read -r in } { err+="$in" }
         print -r ${(qq)err} }
exec {e0}<&p {e1}>&p
coproc :

coproc { exec {e0}<&- {e1}>&-;
         while { IFS= read -r in } { err+="$in" }
	 print -r ${(qq)err} }
exec {o0}<&p {o1}>&p
coproc :

{ print THIS IS STDOUT; print -u2 THIS IS STDERR } >&$o1 2>&$e1
exec {o1}>&- {e1}>&-

# Sadly, read from the two coproc outputs STILL hangs forever
out="$(cat <&$o0)"
err="$(cat <&$e0)"

exec {o0}<&- {e0}<&-

## END SECOND EXAMPLE SCRIPT

So what's going on?  Why doesn't the read loop see end-of-file?

torch% coproc { IFS= read -r in }
[1] 17673
torch% lsof -p $! | grep '[0-9][rwu]'
zsh     17673 schaefer    0r  FIFO    0,7          9175667 pipe
zsh     17673 schaefer    1w  FIFO    0,7          9175666 pipe
zsh     17673 schaefer    2u   CHR  136,6                8 /dev/pts/6
zsh     17673 schaefer   10u   CHR  136,6                8 /dev/pts/6
zsh     17673 schaefer   11r  FIFO    0,7          9175666 pipe
zsh     17673 schaefer   14w  FIFO    0,7          9175667 pipe

What?  Why is the coproc shell holding open the opposite ends of both
ends of the pipe (fd 0r == fd 14w, fd 1w == fd 11r)?  This means this
shell is talking to itself, in effect; it's impossible for it to ever
get EOF on its standard input.

If I make a similar examination of pdksh:

ksh     17650 schaefer    0r  FIFO    0,7         9175095 pipe
ksh     17650 schaefer    1w  FIFO    0,7         9175096 pipe
ksh     17650 schaefer    2u   CHR  136,6               8 /dev/pts/6
ksh     17650 schaefer   12r  FIFO    0,7         9172900 pipe
ksh     17650 schaefer   14r  FIFO    0,7         9172926 pipe

I'm not sure what 12r and 14r represent here, but they're not the same
as 0r, so my second example would work in pdksh with appropriate syntax
adjustments.

If I do this instead:

torch% coproc { exec zsh -fc 'IFS= read -r in' }
[1] 17718
torch% lsof -p $! | grep '[0-9][rwu]'
zsh     17718 schaefer    0r  FIFO    0,7          9176345 pipe
zsh     17718 schaefer    1w  FIFO    0,7          9176344 pipe
zsh     17718 schaefer    2u   CHR  136,6                8 /dev/pts/6
zsh     17718 schaefer   10u   CHR    5,0             2285 /dev/tty
zsh     17718 schaefer   11r   CHR    1,3             2286 /dev/null

Ah, much better.  So those extra copies of the "wrong end" of the pipe
are closed on exec ... but for current shell constructs they ought to
be getting explicitly closed in the subshell run for coproc.

This probably relates back to the discussion from last February that
includes zsh-workers/28762.