Re: reading a file into an array. mapfile? (f)?

[Peter Stephenson <pws@xxxxxxx>]

On Thu, 18 Sep 2008 06:03:38 -0400
"Rocky Bernstein" <rocky.bernstein@xxxxxxxxx> wrote:
> Given this, I find this wording in  zshmodules a little misleading:
> 
>        Thus  it should not automatically be assumed that use of mapfile
> repre‐
>        sents a gain in efficiency over use of other mechanisms.
> 
> Ok. I won't assume it; I will just make use of its speedup over a read loop.

Yes, it more or less is guaranteed to be vastly faster than some other
mechanisms.  What I really meant was `don't assume it's faster than
"$(<filename)"', but it doesn't really need to say that.

> Before posting I tried googling for this and didn't turn up anything. Since
> this is so simple and I think common (perhaps more common that the case
> where one a file as a single long string) possibly this might be mentioned
> in the mapfile doc?

Yes, I think so.

By the way, I'm happy to get partial patches which have been modified to
say something like "say something about X HERE", which helps me locate
where a change is necessary even if the text isn't complete.

> I sort of agree with this comment in zshmodules:
>        It  is  unfortunate that the mechanism for loading modules does not
> yet
>        allow the user to specify the name of the shell parameter to  be
> given
>        the special behaviour.

We've got a better interface to modules now, so we can pass down extra
information with "zmodload -F".  However, it needs to be done carefully:
it's possible different functions would want to map the behaviour onto
different variables.  That's not impossible but would need thought.

It should probably have been called zsh_mapfile to extend the name space in
a more natural way.  I can easily (as with the builtin zstat in zsh/stat)
make the module provide the variable under two different names, so by
default you get mapfile and zsh_mapfile but you could arrange to get only
zsh_mapfile.

> Any thoughts on how to get checksum information? I can shell out to "sum" or
> "md5sum". But given I have the file data as a string if there is a solution
> usesi zsh only, that is preferable.

It sounds like this would need to be a new module in order to do it
internally, with some configuration probing for appropriate libraries using
Clint's new system where the libraries only become dependencies for the
module itself.  It looks like openssl provides this.  You can do things
like "md5sum <<<$file_contents", but there's no real gain over using the
file.  On the other hand, cryptographic functions may typically be
intensive enough that running them inside the shell isn't much of a gain.

Index: Doc/Zsh/mod_mapfile.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/mod_mapfile.yo,v
retrieving revision 1.2
diff -u -r1.2 mod_mapfile.yo
--- Doc/Zsh/mod_mapfile.yo	3 Jul 2007 17:06:04 -0000	1.2
+++ Doc/Zsh/mod_mapfile.yo	18 Sep 2008 10:29:54 -0000
@@ -25,15 +25,18 @@
 
 The parameter tt(mapfile) may be made read-only; in that case, files
 referenced may not be written or deleted.
+
+A file may conveniently be read into an array as one line per element
+with the form `var(array)tt(=LPAR()${(f)mapfile[)var(filename)tt(]RPAR())'.
 )
 enditem()
 
 subsect(Limitations)
 
 Although reading and writing of the file in question is efficiently
-handled, zsh's internal memory management may be arbitrarily baroque.  Thus
-it should not automatically be assumed that use of tt(mapfile) represents a
-gain in efficiency over use of other mechanisms.  Note in particular that
+handled, zsh's internal memory management may be arbitrarily baroque;
+however, tt(mapfile) is usually very much more efficient than
+anything involving a loop.  Note in particular that
 the whole contents of the file will always reside physically in memory when
 accessed (possibly multiple times, due to standard parameter substitution
 operations).  In particular, this means handling of sufficiently long files


-- 
Peter Stephenson <pws@xxxxxxx>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070