Re: line number reporting weirdnesses

["Rocky Bernstein" <rocky.bernstein@xxxxxxxxx>]

Many thanks! All of the tests I've run work great and it fixes a
number of the other problems I hadn't isolated yet but apparently were
the same thing.

I'll add on the debugger side some line number tests. (Actually, it's
kind of hard to do any integration testing and not have it check line
numbers :-)

What you report regarding saving the line number of the first token is
pretty much what I had in mind and have used in other situations.

Personally although there is a lot of complexity in getting the line
number correct, I do think it will help all users in subtle ways such
as in more accurate error reporting or set -x tracing and so on.

Thanks again.

On Sun, Aug 31, 2008 at 3:46 PM, Peter Stephenson
<p.w.stephenson@xxxxxxxxxxxx> wrote:
> On Sun, 31 Aug 2008 16:55:18 +0100
> Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx> wrote:
>> On Sat, 30 Aug 2008 10:49:03 -0400
>> "Rocky Bernstein" <rocky.bernstein@xxxxxxxxx> wrote:
>> > Second the line number in an assignment statement is basically the
>> > line of the string token which spans several lines.  Possibly it might
>> > be more helpful to give the line number of the LHS token.
>>
>> I agree, but that means we need to remember the line number at the start
>> of the last parsed token instead of after.
>
> Circumstantial evidence suggests this can be done in pretty much the
> obvious way.  I've more or less convinced myself that the new variable
> should be valid every time we grab a token and that we can't build up a
> parse tree without doing that, in which case this does the trick.
> I confirmed interactively that it's necessary to save and restore
> toklineno on the lexical stack:  you can get recursive calls when
> parsing strings.  Unfortunately I don't know of a non-interactive test
> for this.
>
> As well as adding a test for the main feature, I've moved all debug tests
> into a different test file.  This saves me about 10 seconds every time I
> run it.
>
> Your experiences with the debugger are almost certainly the best test
> for this beyond the test suite, so I'll commit this as soon as I get
> this message back.
>
> Index: Src/lex.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/lex.c,v
> retrieving revision 1.45
> diff -u -r1.45 lex.c
> --- Src/lex.c   8 Mar 2008 01:20:50 -0000       1.45
> +++ Src/lex.c   31 Aug 2008 19:43:41 -0000
> @@ -46,6 +46,17 @@
>  /**/
>  mod_export int tokfd;
>
> +/*
> + * Line number at which the first character of a token was found.
> + * We always set this in gettok(), which is always called from
> + * yylex() unless we have reached an error.  So it is always
> + * valid when parsing.  It is not useful during execution
> + * of the parsed structure.
> + */
> +
> +/**/
> +zlong toklineno;
> +
>  /* lexical analyzer error flag */
>
>  /**/
> @@ -211,6 +222,7 @@
>
>     unsigned char *cstack;
>     int csp;
> +    zlong toklineno;
>  };
>
>  static struct lexstack *lstack = NULL;
> @@ -269,6 +281,7 @@
>     ls->ecsoffs = ecsoffs;
>     ls->ecssub = ecssub;
>     ls->ecnfunc = ecnfunc;
> +    ls->toklineno = toklineno;
>     cmdsp = 0;
>     inredir = 0;
>     hdocs = NULL;
> @@ -333,6 +346,7 @@
>     ecssub = lstack->ecssub;
>     ecnfunc = lstack->ecnfunc;
>     hlinesz = lstack->hlinesz;
> +    toklineno = lstack->toklineno;
>     errflag = 0;
>
>     ln = lstack->next;
> @@ -661,6 +675,7 @@
>   beginning:
>     tokstr = NULL;
>     while (iblank(c = hgetc()) && !lexstop);
> +    toklineno = lineno;
>     if (lexstop)
>        return (errflag) ? LEXERR : ENDINPUT;
>     isfirstln = 0;
> Index: Src/parse.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
> retrieving revision 1.71
> diff -u -r1.71 parse.c
> --- Src/parse.c 11 Aug 2008 19:22:54 -0000      1.71
> +++ Src/parse.c 31 Aug 2008 19:43:42 -0000
> @@ -721,7 +721,7 @@
>  par_pline(int *complex)
>  {
>     int p;
> -    zlong line = lineno;
> +    zlong line = toklineno;
>
>     p = ecadd(0);
>
> Index: Test/.distfiles
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Test/.distfiles,v
> retrieving revision 1.23
> diff -u -r1.23 .distfiles
> --- Test/.distfiles     31 Aug 2008 13:35:28 -0000      1.23
> +++ Test/.distfiles     31 Aug 2008 19:43:43 -0000
> @@ -17,6 +17,7 @@
>  C02cond.ztst
>  C03traps.ztst
>  C04funcdef.ztst
> +C05debug.ztst
>  D01prompt.ztst
>  D02glob.ztst
>  D03procsubst.ztst
> Index: Test/C03traps.ztst
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Test/C03traps.ztst,v
> retrieving revision 1.16
> diff -u -r1.16 C03traps.ztst
> --- Test/C03traps.ztst  31 Aug 2008 16:01:16 -0000      1.16
> +++ Test/C03traps.ztst  31 Aug 2008 19:43:43 -0000
> @@ -350,94 +350,6 @@
>  >trap
>  >Working 0
>
> -  unsetopt DEBUG_BEFORE_CMD
> -  debug-trap-bug1() {
> -    setopt localtraps
> -    print "print bug file here" >bug-file
> -    print "print this is line one
> -    print this is line two
> -    print this is line three
> -    print and this is line fifty-nine." >bug-file2
> -    function debug_trap_handler {
> -       print $functrace[1]
> -       do_bug
> -    }
> -    function do_bug {
> -       . ./bug-file
> -    }
> -    trap 'echo EXIT hit' EXIT
> -    trap 'debug_trap_handler' DEBUG
> -    . ./bug-file2
> -  }
> -  debug-trap-bug1
> -0: Relationship between traps and sources
> ->debug-trap-bug1:15
> ->bug file here
> ->this is line one
> ->./bug-file2:1
> ->bug file here
> ->this is line two
> ->./bug-file2:2
> ->bug file here
> ->this is line three
> ->./bug-file2:3
> ->bug file here
> ->and this is line fifty-nine.
> ->./bug-file2:4
> ->bug file here
> ->debug-trap-bug1:16
> ->bug file here
> ->EXIT hit
> -
> -  cat >zsh-trapreturn-bug2 <<-'HERE'
> -       cmd='./fdasfsdafd'
> -       [[ -x $cmd ]] && rm $cmd
> -       set -o DEBUG_BEFORE_CMD
> -       trap '[[ $? -ne 0 ]] && exit 0' DEBUG
> -       $cmd  # invalid command
> -       # Failure
> -       exit 10
> -       HERE
> -  $ZTST_testdir/../Src/zsh -f ./zsh-trapreturn-bug2
> -0: trapreturn handling bug is properly fixed
> -?./zsh-trapreturn-bug2:5: no such file or directory: ./fdasfsdafd
> -
> -  fn() {
> -    setopt localtraps localoptions debugbeforecmd
> -    trap '(( LINENO == 4 )) && setopt errexit' DEBUG
> -    print $LINENO three
> -    print $LINENO four
> -    print $LINENO five
> -    [[ -o errexit ]] && print "Hey, ERREXIT is set!"
> -  }
> -  fn
> -1:Skip line from DEBUG trap
> ->3 three
> ->5 five
> -
> -  # Assignments are a special case, since they use a simpler
> -  # wordcode type, so we need to test skipping them separately.
> -  fn() {
> -    setopt localtraps localoptions debugbeforecmd
> -    trap '(( LINENO == 4 )) && setopt errexit' DEBUG
> -    x=three
> -    x=four
> -    print $LINENO $x
> -    [[ -o errexit ]] && print "Hey, ERREXIT is set!"
> -  }
> -  fn
> -1:Skip assignment from DEBUG trap
> ->5 three
> -
> -  fn() {
> -    setopt localtraps localoptions debugbeforecmd
> -    trap 'print $LINENO' DEBUG
> -    [[ a = a ]] && print a is ok
> -  }
> -  fn
> -0:line numbers of complex sublists
> ->3
> ->a is ok
>
>  %clean
>
> Index: Test/C05debug.ztst
> ===================================================================
> RCS file: Test/C05debug.ztst
> diff -N Test/C05debug.ztst
> --- /dev/null   1 Jan 1970 00:00:00 -0000
> +++ Test/C05debug.ztst  31 Aug 2008 19:43:43 -0000
> @@ -0,0 +1,114 @@
> +%prep
> +
> +  setopt localtraps
> +
> +%test
> +
> +  unsetopt DEBUG_BEFORE_CMD
> +  debug-trap-bug1() {
> +    setopt localtraps
> +    print "print bug file here" >bug-file
> +    print "print this is line one
> +    print this is line two
> +    print this is line three
> +    print and this is line fifty-nine." >bug-file2
> +    function debug_trap_handler {
> +       print $functrace[1]
> +       do_bug
> +    }
> +    function do_bug {
> +       . ./bug-file
> +    }
> +    trap 'echo EXIT hit' EXIT
> +    trap 'debug_trap_handler' DEBUG
> +    . ./bug-file2
> +  }
> +  debug-trap-bug1
> +0: Relationship between traps and sources
> +>debug-trap-bug1:15
> +>bug file here
> +>this is line one
> +>./bug-file2:1
> +>bug file here
> +>this is line two
> +>./bug-file2:2
> +>bug file here
> +>this is line three
> +>./bug-file2:3
> +>bug file here
> +>and this is line fifty-nine.
> +>./bug-file2:4
> +>bug file here
> +>debug-trap-bug1:16
> +>bug file here
> +>EXIT hit
> +
> +  cat >zsh-trapreturn-bug2 <<-'HERE'
> +       cmd='./fdasfsdafd'
> +       [[ -x $cmd ]] && rm $cmd
> +       set -o DEBUG_BEFORE_CMD
> +       trap '[[ $? -ne 0 ]] && exit 0' DEBUG
> +       $cmd  # invalid command
> +       # Failure
> +       exit 10
> +       HERE
> +  $ZTST_testdir/../Src/zsh -f ./zsh-trapreturn-bug2
> +0: trapreturn handling bug is properly fixed
> +?./zsh-trapreturn-bug2:5: no such file or directory: ./fdasfsdafd
> +
> +  fn() {
> +    setopt localtraps localoptions debugbeforecmd
> +    trap '(( LINENO == 4 )) && setopt errexit' DEBUG
> +    print $LINENO three
> +    print $LINENO four
> +    print $LINENO five
> +    [[ -o errexit ]] && print "Hey, ERREXIT is set!"
> +  }
> +  fn
> +1:Skip line from DEBUG trap
> +>3 three
> +>5 five
> +
> +  # Assignments are a special case, since they use a simpler
> +  # wordcode type, so we need to test skipping them separately.
> +  fn() {
> +    setopt localtraps localoptions debugbeforecmd
> +    trap '(( LINENO == 4 )) && setopt errexit' DEBUG
> +    x=three
> +    x=four
> +    print $LINENO $x
> +    [[ -o errexit ]] && print "Hey, ERREXIT is set!"
> +  }
> +  fn
> +1:Skip assignment from DEBUG trap
> +>5 three
> +
> +  fn() {
> +    setopt localtraps localoptions debugbeforecmd
> +    trap 'print $LINENO' DEBUG
> +    [[ a = a ]] && print a is ok
> +  }
> +  fn
> +0:line numbers of complex sublists
> +>3
> +>a is ok
> +
> +  fn() {
> +    setopt localtraps localoptions debugbeforecmd
> +    trap 'print $LINENO' DEBUG
> +    print before
> +    x='    first
> +    second
> +    third'
> +    print $x
> +  }
> +  fn
> +0:line numbers of multiline assignments
> +>3
> +>before
> +>4
> +>7
> +>    first
> +>    second
> +>    third
> +
>
> --
> Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>
> Web page now at http://homepage.ntlworld.com/p.w.stephenson/
>