Re: [PATCH v2] exec: run final pipeline command in a subshell in sh modeZZ

[Daniel Shahaf <d.s@xxxxxxxxxxxxxxxxxx>]

brian m. carlson wrote on Fri, 05 Jun 2020 20:41 +0000:
> On 2020-06-05 at 10:21:41, Mikael Magnusson wrote:
> > On 6/5/20, brian m. carlson <sandals@xxxxxxxxxxxxxxxxxxxx> wrote:  
> > > zsh typically runs the final command in a pipeline in the main shell
> > > instead of a subshell.  However, POSIX requires that all commands in a
> > > pipeline run in a subshell, but permits zsh's behavior as an extension.  
> > 
> > What POSIX actually says is:
> > "each command of a multi-command pipeline is in a subshell
> > environment; as an extension, however, any or all commands in a
> > pipeline may be executed in the current environment"

That's quoted from https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12.

The part Brian quotes below is from https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap02.html#tag_02_01_01.

> > Ie, it does not say "shall", so it doesn't require a subshell all, in
> > fact it explicitly does permit not using one as you also say. The

This interpretation is analogous to how conforming C programs must
assume neither that «char» is signed nor that it is unsigned.

> > patch is possibly useful (seems unlikely to me), but to say it is
> > required by POSIX is not true. If someone depends on every command in
> > a pipeline being a subshell, they should fix their code, for example
> > by adding ( ) around it (the command(s) or the whole pipeline).  
> 
> POSIX makes a declarative statement about the behavior of a pipeline.
> It is true that it doesn't explicitly use the word "shall" in this case,
> since such a statement would explicitly prohibit the inclusion of an
> extension at all and make it explicitly non-conforming.
> 

The sentence preceding the one you quoted reads:
.
    Non-standard extensions, when used, may change the behavior of
    utilities, functions, or facilities defined by POSIX.1-2017.

I take this to mean non-standard extensions aren't bound by "shall"s.

As to why the passage Mikael quoted doesn't use the word "shall"… well,
presumably it doesn't use the word "shall" because it doesn't describe
"a feature or behavior that is mandatory"¹.

> What POSIX does say is that one “shall define an environment in which an
> application can be run with the behavior specified by POSIX.1-2017.”
> I'm proposing that "zsh --emulate sh" implement the POSIX behavior for
> that reason.

What Mikael's saying is that zsh's incumbent behaviour is already
POSIX-conforming, but POSIX-conforming implementations have some leeway:
have a range of possible behaviours to choose from, just like conforming
C compilers can choose what signedness to give to «char».

The passage Mikael quoted specifies that running the last command in
a pipeline in a subshell by default is permitted in certain cases,
outlined by the phrases "as an extension" and "may".

The definition of "may"¹ says it's used to describe "optional" behaviours,
and that conforming applications should tolerate both presence and
absence of that behaviour.

As to that behaviour's being an extension, the sentence you quoted, once
put back in context (link given above), requires conforming implementations
to document how to disable non-standard extensions where they change
POSIX-documented behaviour.  I don't think the passage Mikael quoted is
covered by that: it _is_ an extension, of course, but I question whether
it's a "non-standard extension"².  After all, it is specified in the
standard.

To summarize, I don't see why behaviour specified with the phrases "as
an extension" and "may" should be off by default in a POSIX-conforming
mode.  Would you elaborate on this?

(On the other hand, I'm not sure why they bothered to write the words
"as an extension" there.  They don't seem to change the meaning one way
or the other.)

> I will tell you that as a practical matter, nobody writing code for sh
> expects the last command not to be run in a subshell and consequently
> lots of code is practically broken in this case with zsh as /bin/sh.
> The Git Project is very fastidious about writing portable shell, as is
> Debian, and I can tell you from experience that both are broken with zsh
> as sh with the current behavior, even if they should not have made that
> assumption.
> 
> I would also like to see macOS users who decide to use zsh as /bin/sh
> have a good experience with existing code that overwhelmingly does
> make this assumption.

Well, perhaps there is something we can do to make their lives easier.

Continuing the analogy to C, gcc(1) has -fsigned-char/-funsigned-char
flags to help unportable programs.  However, I hesitate to propose
adding an option just for this: adding options is always easy to
suggest, but not always a good idea.

Since zsh already incorporates a parser for sh scripts, perhaps we could
write a tool that automatically adds parentheses to the last element in
every pipeline.  That's not such a crazy idea: it already exists (in a
much more general form) for C: http://coccinelle.lip6.fr/

[Between these two, an extra option is definitely the lower-hanging
fruit, of course.]

> zsh is a very popular interactive shell, and allowing it to be used as a
> portable sh on systems where the system sh is less capable would be
> really beneficial.

How would it be beneficial?

> If your objection is to the wording, I'm happy to revise it to remove
> the word "requires", but I do think this provides a lot of benefits for
> the sh scripting case while not impacting users who are expecting
> different behavior for the zsh case.

The patch would constitute a backwards-incompatible change to anyone who
uses zsh as sh today and relies on the current behaviour of pipelines.

This might have been acceptable if it were a question of changing
a non-conforming behaviour to a conforming behaviour.  However, the
current behaviour does appear to be conforming.

Furthermore, if the patch is accepted, those who rely on the incumbent
behaviour won't have an easy workaround to get it behaviour back,
something comparable to the "add parentheses around every element of
every pipeline" strategy that can be used given the incumbent
implementation to get the patch's semantics.

Cheers,

Daniel

¹ https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap01.html

² If POSIX didn't want to make this distinction, it would have written
just "extension", unqualified.