Re: Parsing CVS files

[Peter Stephenson <p.w.stephenson@xxxxxxxxxxxx>]

> I'm looking for an easy way to split the lines of a .csv file into
> the fields of an array variable.  There's a script that does that
> somewhore on the net.  But that script parses lines character by
> character and just manages to parse about 100 (long) lines per
> second.
> 
> Fields in a .csv file are separated by commas, *but* commas
> between a pair of quotes do not split.  Or phrased differently:
> Commas that have an even number of double quotes left of them do
> split, but commas with an uneven number left of then don't split.
> 
> Any ideas for a quick implementation?

Sebastian has done similar things so may have better ideas.

If you're happy to use shell syntax --- in other words, the other forms
of quoting are active, not just double quotes, so backslashes and single
quotes might do inconvenient things --- and you're not too bothered
about unquoted spaces, which will add extra lines of splitting, you can
use this trick:

% line='This,"is, quite possibly, a",line,"of,stuff","with,commas"'
% print -rl ${(Q)${${(z)${line//,/, }}%%,}//, /,}
This
is, quite possibly, a
line
of,stuff
with,commas

Each comma gets a space added, then the line is split on syntactically
active spaces; any comma at the end of a field is removed; the remaining
commas are restored.

To strip the quotes, add the (Q) flag to the outermost step.

If you need to be careful about unquoted spaces, you need to be
cleverer:  e.g. backslash quote them and then remove the bacslashes
later.  E.g. up to subtle effects associated with backslashes

print -rl ${${${${(z)${${line// /\\ }//,/, }}%%,}//, /,}//\\ / }

will retain existing spaces.

Also, if you want to keep empty fields, you'll need the final result
to use "${(@}this}".  Probably easiest to assign to an array as otherwise
the quotes will affect the substitution.

If you're worried about subtle effects with backslashes, I don't think
you're ever going to be satisfied with a quick and dirty hack like this,
so you'll have to decide how sophisticated you need to be.

pws