Re: Extended globbing seems to have become much slower in recent versions of Zsh

[Jesper Nygårds <jesper.nygards@xxxxxxxxx>]

I don't know if this is relevant, but I have some more findings. I wanted
to know which sub directory was contributing the most to the amount of time
taken to process the root directory. I then realized that the sum of the
time it took to process each sub directory separately was much lower than
processing the whole root directory at once. In the run below, you can see
that whilst it takes about 36 seconds to process the root directory, it
only takes about 13 seconds to process all directories one at a time.
Furthermore, when I descend into the sub directory taking the longest time
in the first run, and run all its sub directories in sequence, the sum of
these sub-sub directories is significantly lower than for the whole sub
directory. So obviously the processing time is not linear with the number
of files.

% float SECONDS MYNOW;

% MYNOW=$SECONDS; :
(#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
print $(( $SECONDS - $MYNOW ))
35.810789000000113

% for dir in */; do print $dir; cd $dir; MYNOW=$SECONDS; :
(#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
print $(( $SECONDS - $MYNOW )); cd -q - >|/dev/null; done
common-pom/
0.00011700000004566391
common.wiki/
0.00033800000028350041
common/
1.0273979999997209
intygstjanst-konfiguration/
0.00071800000023358734
intygstjanst/
0.08040000000028158
intygstyper/
9.1739539999998669
minaintyg-konfiguration/
0.00063700000009703217
minaintyg/
0.67290499999990061
privatlakarportal-konfiguration/
0.0015870000001996232
privatlakarportal/
0.83380199999965043
schemas/
0.11772999999993772
tools/
0.0098799999996117549
webcert-konfiguration/
0.0018289999998160056
webcert/
1.4324129999999968

% cd intygstyper

% for dir in */; do print $dir; cd $dir; MYNOW=$SECONDS; :
(#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
print $(( $SECONDS - $MYNOW )); cd -q - >|/dev/null; done
fk/
1.9322819999997591
fk7263/
0.37481900000011592
target/
0.17797399999972185
tools/
0.0080309999998462445
ts-bas/
0.31286899999986417
ts-diabetes/
0.22946499999989101
ts-parent/
0.0033199999998032581


On Fri, Mar 4, 2016 at 9:17 AM, Jesper Nygårds <jesper.nygards@xxxxxxxxx>
wrote:

> I have run 5.2, compiled with and without the --with-stack-allocation
> flag. It made no difference however. You can see that the runs below are
> very similar in the measured times.
>
> With "--with-stack-allocation":
>
> % zsh --version
> zsh 5.2 (x86_64-apple-darwin14.5.0)
>
> % /usr/local/bin/zsh -f
> % setopt EXTENDED_GLOB; float SECONDS MYNOW
>
> % MYNOW=$SECONDS; :
> **/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*)(-.D);
> print $(( $SECONDS - $MYNOW ))
> 54.868589
>
> % MYNOW=$SECONDS; :
> (#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*)(-.D);
> print $(( $SECONDS - $MYNOW ))
> 47.001608000000019
>
> % MYNOW=$SECONDS; :
> **/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
> print $(( $SECONDS - $MYNOW ))
> 38.805462000000006
>
> % MYNOW=$SECONDS; :
> (#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
> print $(( $SECONDS - $MYNOW ))
> 32.296016000000009
>
> -------------------------------------------------------
>
> Without "--with-stack-allocation":
>
> % /usr/local/bin/zsh --version
> zsh 5.2 (x86_64-apple-darwin14.5.0)
>
> % /usr/local/bin/zsh -f
> % setopt EXTENDED_GLOB; float SECONDS MYNOW
>
> % MYNOW=$SECONDS; :
> **/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*)(-.D);
> print $(( $SECONDS - $MYNOW ))
> 53.115647000000003
>
> % MYNOW=$SECONDS; :
> (#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*)(-.D);
> print $(( $SECONDS - $MYNOW ))
> 42.988532000000006
>
> % MYNOW=$SECONDS; :
> **/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
> print $(( $SECONDS - $MYNOW ))
> 36.619570999999993
>
> % MYNOW=$SECONDS; :
> (#i)**/*~(|*/)(build|target|node|node_modules|.idea|*.min.js|*.iml|TAGS)(|/*);
> print $(( $SECONDS - $MYNOW ))
> 30.786290000000008
>
>
> On Thu, Mar 3, 2016 at 1:06 AM, Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>
> wrote:
>
>> On Wed, Mar 2, 2016 at 12:39 AM, Jesper Nygårds
>> <jesper.nygards@xxxxxxxxx> wrote:
>> > Previously I was using the Apple-provided version of 5.0.5, and the
>> > Homebrew version of 5.2. Now I have instead used different versions of
>> > Homebrew zsh, and I have made sure that the compiler flags were the same
>> > through all versions. I have then run the same tests again, with
>> versions
>> > 5.0.5, 5.0.6, 5.1 and 5.2.
>>
>> OK, thanks; next thing to try:  configure --with-stack-allocation
>>
>> I've scanned through diffs and I just don't see any structural changes
>> that would account for that slowdown; the above is a bit of a
>> longshot.
>>
>> Incidentally (for -workers) there's a typo in ChangeLog -- 32932:
>> Src/glob.c -- should say Src/subst.c.
>>
>
>