Discussion of path search for builtin utilities

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

---------- Forwarded message ----------
Date: Wed, 2 Jul 2014 23:29:29 +0200
From: Jilles Tjoelker <jilles@xxxxxxxx>
To: Philip Guenther <guenther@xxxxxxxxx>
Cc: austin-group-l@xxxxxxxxxxxxx
Subject: Re: what does XCU 2.9.1.1 part 1.d.i.a. mean?  Does anyone follow it?
Resent-Date: 2 Jul 2014 21:35:13 -0000
Resent-From: austin-group-l@xxxxxxxxxxxxx
Resent-To: austin-group-l@xxxxxxxxxxxxx

On Tue, Jul 01, 2014 at 12:54:32AM -0700, Philip Guenther wrote:
> XCU 2.9.1.1 describes how the shell does command seach and execution.  To
> quote:
> ------
> 1. If the command name does not contain any <slash> characters, the first
>    successful step in the following sequence shall occur:
>     a.  If the command name matches the name of a special built-in
>         utility, that special built-in utility shall be invoked.
>     b.  If the command name matches the name of a function known to
>         this shell, the function shall be invoked as described in
>         Section 2.9.5. If the implementation has provided a standard
>         utility in the form of a function, it shall not be recognized
>         at this point. It shall be invoked in conjunction with the path
>         search in step 1.d.
>     c.  If the command name matches the name of a utility listed in the
>         following table, that utility shall be invoked.
>                 alias        false       jobs       read         wait
>                 bg           fc          kill       true
>                 cd           fg          newgrp     umask
>                 command      getopts     pwd        unalias
>     d.  Otherwise, the command shall be searched for using the PATH
>         environment variable as described in XBD Chapter 8:
>      i. If the search is successful:
>         a. If the system has implemented the utility as a regular
>            built-in or as a shell function, it shall be invoked at this
>            point in the path search.
>         b. Otherwise, the shell executes the utility in a separate utility
>            environment (see Section 2.12) with actions equivalent to
>            calling the execl() function as defined in <...>
> ...
>      ii. If the search is unsuccessful, the command shall fail with an
>          exit status of 127 and the shell shall write an error message.
> ------

> Okay, so consider a shell which has 'test' as a built-in.  It's not
> special, let's assume there's no function definition, and its not in the
> list in 1.c, so the shell has to do a search of $PATH per 1.d.

> Question 1)
>    Part 1.d.i.a. *doesn't* refer to the search finding the utility in any
>    particular directory, such as a directory in the PATH required for
>    standards compliance.  So it doesn't matter if it found a completely
>    different version of the utility; it can/should still invoke the
>    (standard-behaving) built-in version.  Is that correct?

> Question 2)
>    Part 1.d.ii says that if you override PATH with one that doesn't
>    include (versions of) the standard utilties, then built-ins not listed
>    in part 1c must not be invoked by the shell.  For example, assuming
>    $HOME/empty doesn't exist and can be created, then this:
> 	mkdir $HOME/empty
> 	( unset -f test; PATH=$HOME/empty; test -d $HOME/empty )
> 	echo $?
>    must output "127".  Is that correct?

> Checking of a few systems (linux, solaris 10, Mac OSX) find they seem to
> answer my questions 1 and 2 with "yes" and "no", respectively.  That is,
> they treat all built-ins like the list in 1c, skipping the $PATH search
> completely and rendering part 1.d.i.a. superfluous.

> (If it matter neither whether the path search is successful, nor what it
> finds when it is successful, then the search is pointless.)

> Can we delete part 1.d.i.a ?

ksh93 implements 1.d.i.a for some builtins, although it also has some
builtins (POSIX and other) that are not found via PATH.

For example, ksh93 Version AJM 93u+ 2012-08-01 from FreeBSD ports can do
things like (after creating an executable named /opt/ast/bin/cat):

$ command -V cat
cat is a tracked alias for /bin/cat
$ (PATH=/opt/ast/bin:$PATH; command -V cat)
cat is a shell builtin version of /opt/ast/bin/cat
$ (PATH=/opt/ast/bin:$PATH; cat -\?)
Usage: cat [-bdenstuvABDERST] [file ...]
$ cat -\?
cat: illegal option -- ?
usage: cat [-belnstuv] [file ...]
$ PATH=/var/empty cat -\?
ksh93: cat: not found [No such file or directory]

I think this is what is meant by 1.d.i.a.

This ksh93 feature allows scripts to opt-in to builtins that may not be
compatible with the utilities provided with the system (for example, the
builtin version does not support FreeBSD's -l option). For a shell that
is more tightly coupled with the rest of the system, this seems less
useful.

The apparent goal of allowing applications to be certain of invoking
their own executables is not achieved by almost all shells, and may not
be desirable either (how about calling utilities implemented as shell
scripts using a non-standard PATH?).

-- 
Jilles Tjoelker

---------- Forwarded message ----------
Date: Tue, 1 Jul 2014 12:37:56 -0700
From: Nick Stoughton <nickstoughton@xxxxxxxxx>
To: Philip Guenther <guenther@xxxxxxxxx>
Cc: "austin-group-l@xxxxxxxxxxxxx" <austin-group-l@xxxxxxxxxxxxx>
Subject: Re: what does XCU 2.9.1.1 part 1.d.i.a. mean? Does anyone follow it?
Resent-Date: 1 Jul 2014 19:38:08 -0000
Resent-From: austin-group-l@xxxxxxxxxxxxx
Resent-To: austin-group-l@xxxxxxxxxxxxx

As far as I can see all of the following shells fail to conform as you
describe:

ksh
bash
dash
tcsh
zsh

(some of these fail for lots of other reasons too!)

The only shell I have been able to find that *does* implement the standard
as required is "yash" (and then only in posixly-correct mode).

However, XRAT (page 3693, line 126346 onwards) has this to say:
--------------
The command search shown here does not match all historical
implementations. A more typical
sequence has been:
 ?  Any built-in (special or regular)
 ? Functions
 ? Path search for executable files
But there are problems with this sequence. Since the programmer has no idea
in advance which
utilities might have been built into the shell, a function cannot be used
to override portably a
utility of the same name. (For example, a function named cd cannot be
written for many
historical systems.) Furthermore, the PATH variable is partially
ineffective in this case, and only
a pathname with a <slash> can be used to ensure a specific executable file
is invoked.

After the execve( ) failure described, the shell normally executes the file
as a shell script. Some
implementations, however, attempt to detect whether the file is actually a
script and not an
executable from some other architecture. The method used by the KornShell
is allowed by the
text that indicates non-text files may be bypassed.

The sequence selected for the Shell and Utilities volume of POSIX.1-2008
acknowledges that
special built-ins cannot be overridden, but gives the programmer full
control over which
versions of other utilities are executed. It provides a means of
suppressing function lookup (via
the command utility) for the user?s own functions and ensures that any
regular built-ins or
functions provided by the implementation are under the control of the path
search. The
mechanisms for associating built-ins or functions with executable files in
the path are not
specified by the Shell and Utilities volume of POSIX.1-2008, but the
wording requires that if
either is implemented, the application is not able to distinguish a
function or built-in from an
executable (other than in terms of performance, presumably). The
implementation ensures that
all effects specified by the Shell and Utilities volume of POSIX.1-2008
resulting from the
invocation of the regular built-in or function (interaction with the
environment, variables, traps,
and so on) are identical to those resulting from the invocation of an
executable file.

------------
So it appears that this is deliberate, if largely un-implemented. I would
encourage you to file a bug at http://austingroupbugs.net for further
deliberation.