Re: Hugh number of file descriptor checks

[Phil Pennock <zsh-workers+phil.pennock@xxxxxxxxxxxx>]

On 2010-02-19 at 00:00 +0100, Joke de Buhr wrote:
> I did some system call traces of a c program while I discovered zsh doing a 
> large amount of file descriptor checks during startup which seem unnecessary.
> 
> Doing a simple strace like "strace zsh -c 'echo hello'" produces an output of 
> about 4280 system calls which include a sequence of 4032 system calls like:
> 
>     fcntl(64, F_GETFL)      = -1 EBADF (Bad file descriptor)
>     fcntl(65, F_GETFL)      = -1 EBADF (Bad file descriptor)

This appears to be zopenmax():
----------------------------8< cut here >8------------------------------
#ifdef HAVE_SYSCONF
/* This is replaced by a macro from system.h if not HAVE_SYSCONF.    *
 * 0 is returned by sysconf if _SC_OPEN_MAX is unavailable;          *
 * -1 is returned on error                                           *
 *                                                                   *
 * Neither of these should happen, but resort to OPEN_MAX rather     *
 * than return 0 or -1 just in case.                                 */

/**/
mod_export long
zopenmax(void)
{
    static long openmax = 0;

    if (openmax < 1) {
        if ((openmax = sysconf(_SC_OPEN_MAX)) < 1) {
            openmax = OPEN_MAX;
        } else if (openmax > OPEN_MAX) {
            /* On some systems, "limit descriptors unlimited" or the  *
             * equivalent will set openmax to a huge number.  Unless  *
             * there actually is a file descriptor > OPEN_MAX already *
             * open, nothing in zsh requires the true maximum, and in *
             * fact it causes inefficiency elsewhere if we report it. *
             * So, report the maximum of OPEN_MAX or the largest open *
             * descriptor (is there a better way to find that?).      */
            long i, j = OPEN_MAX;
            for (i = j; i < openmax; i += (errno != EINTR)) {
                errno = 0;
                if (fcntl(i, F_GETFL, 0) < 0 &&
                    (errno == EBADF || errno == EINTR))
                    continue;
                j = i;
            }
            openmax = j;
        }
    }

    return (max_zsh_fd > openmax) ? max_zsh_fd : openmax;
}
#endif
----------------------------8< cut here >8------------------------------

On FreeBSD, sysconf(_SC_OPEN_MAX) calls getrlimit(RLIMIT_NOFILE); if
that fails, or is RLIM_INFINITY or is larger than LONG_MAX, then it
returns -1.  Otherwise, it returns the current rlimit value.

So on my box, that's 11095.  Meanwhile, OPEN_MAX is 64.

SUSv3 documents:
----------------------------8< cut here >8------------------------------
{OPEN_MAX}
  Maximum number of files that one process can have open at any one time.
  Minimum Acceptable Value: {_POSIX_OPEN_MAX}
{_POSIX_OPEN_MAX}
  Maximum number of files that one process can have open at any one time.
  Value: 20
----------------------------8< cut here >8------------------------------

Meanwhile, max_zsh_fd is the largest fd that has been allocated by the
shell itself, rather than inherited.

zopenmax() is used to close all file-descriptors except some which are
being kept, and to determine the size of the fdtable which is nominally
what sets max_zsh_fd.  zopenmax() returns OPEN_MAX in the event that
sysconf() isn't available.


Questions for the list:
 Why not just pick a magic value like 64 or 32 and, if
 sysconf(_SC_OPEN_MAX) is larger than that, use the magic value?  Why do
 we need to know the largest fd already in use?  Does fdtable have to be
 able to handle the largest fd already open, if not opened by zsh?

-Phil