Re: zsh and memory

[Sven Wischnowsky <wischnow@xxxxxxxxxxxxxxxxxxxxxxx>]

[ Moved to zsh-workers ]

Bart Schaefer wrote:

> On Dec 13,  6:15am, Danny Dulai wrote:
> } Subject: zsh and memory
> }
> } Any hints on how to make zsh not consume as much memory as it does?
> } I'm not using the zftp or stat modules. Any thing else I can do?
> 
> It'd be easier to answer the question if we knew what you ARE doing.
> "Is there any way I can get rid of these lumps on my head?  I'm not
> hitting myself with a hammer."
> 
> Various things you should avoid if memory is a problem include:
> 
>     setopt ALLEXPORT
>     use large here-documents
>     define lots of shell functions without using autoload
>     use "zed" (old) or the mapfile module (new) to edit files
>     use the new 3.1.6 shell-function-based completion system
> 
> The last one is a bit unfortunate, but just "compinit" (without even
> trying any completions yet) adds half a megabyte to the RSS of zsh on
> my system, and it only goes up from there as functions autload and
> start caching their results in shell variables.

I've had a look at this... Horrors!


I tried it with `zsh -f', with `zsh -f' and loading the completion
system, and with that and doing `ls ../Com/Cor<TAB>'. Looking at the
output of the mem builtin first revealed that there were almost 400KB
of free memory, in relatively few large blocks -- remnents of freed
heaps. So I looked who was using lots of heap memory and found the
tokstr handling in lex.c. There we always allocate at least 256 bytes
and if the buffer needs expanding (in add()) in gets resized to
inbufct (or larger). Some more investigation showed that we almost
never need a tokstr with more than 32 bytes, so I changed that. I also 
changed add() to be more careful when expanding the buffer -- I left
the old code conditioned out because I don't kno if there was a reason 
to resize it to inbufct bytes; I at least don't see a reason for that.

Then I had another look at the mem output and had to see several
*used* blocks starting with `#autoload..# ...'. Yes, indeed, the
beginnings of the autoloaded functions. getfpfunc() uses a zalloc()ed
buffer for the whole file content and didn't free it (it may be that
this was my fault, may have happened in the non-copying execution code 
patches). So I fixed that, too.

Then I added pushheap()s and popheap()s around function loading code
and that helped some more.

These patches save me (in the tests mentioned above) 200KB and 600KB,
respectively (resident size). But I still have about 160KB of free
memory (old heaps). I'm really tempted to allocate heaps using mmap()
(anonymous) to get them out of the way of the zalloc() allocator. I
small test showed that with this I only get 39KB of free memory after
the completion test, which is really not too bad. I don't have a patch 
for that yet, though.

Bye
 Sven

diff -ru ../z.old/Src/exec.c Src/exec.c
--- ../z.old/Src/exec.c	Thu Dec 16 09:46:11 1999
+++ Src/exec.c	Thu Dec 16 10:10:07 1999
@@ -2905,6 +2905,8 @@
     int noalias = noaliases;
     List l;
 
+    pushheap();
+
     noaliases = (shf->flags & PM_UNALIASED);
     l = getfpfunc(shf->nam);
     noaliases = noalias;
@@ -2929,6 +2931,8 @@
 	} LASTALLOC;
 	shf->flags &= ~PM_UNDEFINED;
     }
+    popheap();
+
     execlist(shf->funcdef, 1, 0);
     return lastval;
 }
@@ -2942,6 +2946,8 @@
     int noalias = noaliases;
     List l;
 
+    pushheap();
+
     noaliases = (shf->flags & PM_UNALIASED);
     l = getfpfunc(shf->nam);
     noaliases = noalias;
@@ -2955,6 +2961,8 @@
     } LASTALLOC;
     shf->flags &= ~PM_UNDEFINED;
 
+    popheap();
+
     return 0;
 }
 
@@ -3118,9 +3126,14 @@
 		    HEAPALLOC {
 			r = parse_string(d, 1);
 		    } LASTALLOC;
+
+		    zfree(d, len + 1);
+
 		    return r;
 		} else
 		    close(fd);
+
+		zfree(d, len + 1);
 	    } else {
 		close(fd);
 	    }
diff -ru ../z.old/Src/lex.c Src/lex.c
--- ../z.old/Src/lex.c	Thu Dec 16 09:46:12 1999
+++ Src/lex.c	Thu Dec 16 09:57:18 1999
@@ -470,6 +470,7 @@
 {
     *bptr++ = c;
     if (bsiz == ++len) {
+#if 0
 	int newbsiz;
 
 	newbsiz = bsiz * 8;
@@ -477,6 +478,12 @@
 	    newbsiz *= 2;
 	bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz));
 	bsiz = newbsiz;
+#endif
+
+	int newbsiz = bsiz + 32;
+
+	bptr = len + (tokstr = (char *)hrealloc(tokstr, bsiz, newbsiz));
+	bsiz = newbsiz;
     }
 }
 
@@ -556,7 +563,7 @@
     /* word includes the last character read and possibly \ before ! */
     if (dbparens) {
 	len = 0;
-	bptr = tokstr = (char *)ncalloc(bsiz = 256);
+	bptr = tokstr = (char *)ncalloc(bsiz = 32);
 	hungetc(c);
 	cmdpush(CS_MATH);
 	c = dquote_parse(infor ? ';' : ')', 0);
@@ -671,7 +678,7 @@
 	    }
 	    if (incmdpos) {
 		len = 0;
-		bptr = tokstr = (char *)ncalloc(bsiz = 256);
+		bptr = tokstr = (char *)ncalloc(bsiz = 32);
 		return cmd_or_math(CS_MATH) ? DINPAR : INPAR;
 	    }
 	} else if (d == ')')
@@ -818,7 +825,7 @@
     peek = STRING;
     if (!sub) {
 	len = 0;
-	bptr = tokstr = (char *)ncalloc(bsiz = 256);
+	bptr = tokstr = (char *)ncalloc(bsiz = 32);
     }
     for (;;) {
 	int act;

--
Sven Wischnowsky                         wischnow@xxxxxxxxxxxxxxxxxxxxxxx