Re: Re[2]: High memory usage on // substitution in one situation, normal usage in other

[Bart Schaefer <schaefer@xxxxxxxxxxxxxxxx>]

On May 21, 10:27pm, Manuel Presnitz wrote:
}
} I can reproduce the extraordinary high memory usage ...
} but only if EXTENDEDGLOB is set.

OK, that helps (and I should have caught that this is required for (#b)
to be meaningful).

Let's consider the two examples:

1) arr2 is an associative array mapping 200000 integers to underscores.
   Flattened into a string, you get alternating number-strings and
   underscores separated by spaces.

2) arr is an array of 900000 single underscores.  Flattened it's a
   string of underscores separated by spaces.

Now let's look more closely at the pattern:

    [^$'\03'-$'\07'$'\013'-$'\014'$'\016'-$'\031'$'0\037']##

Look at the rightmost $'...' expression:  $'0\037'

That has a "0" character in it.  Probably a typo, but it means that
every time there is a zero digit in $__text, the // recursion has to
save state before proceeding.  This doesn't happen when the text is
nothng but underscores.  The state that is being saved includes the
entire tail of the string from that point onward, so in the first
example it requires approximately 1.8MB/2*(the number of zeroes in
all integers from 1 to 200000), and in the second case it never does
recur (the ## consumes the entire string).

The memory is being allocated here:

		/*
		 * Array to record the start of characters for
		 * backtracking.
		 */
		VARARR(char, charstart, patinend-patinput);
		memset(charstart, 0, patinend-patinput);

(pattern.c:3263)