Re: Multi-core loops

["Nadav Har'El" <nyh@xxxxxxxxxxxxxxxxxxx>]

On Sun, Nov 22, 2009, Kazuo Teramoto wrote about "Re: Multi-core loops":
> On Sun, Jun 28, 2009 at 8:31 AM, Nadav Har'El <nyh@xxxxxxxxxxxxxxxxxxx> wrote:
> > Zsh, like all shells, lets you easily do something many times in a loop.
>...
> > But when "dosomething" is CPU intensive, this is *not* what you'd want to
> > do on a multi-core (multi CPU) machine, which have more-or-less become
> > standard nowadays...
> > Such a loop would only use one of the CPUs, and leave the other(s) unused.
> > Instead, you'll want to keep all CPUs busy all the time, running M (=number
> > of CPUs) processes at the same time.
> 
> Any update on this?
> 
> I'm searching for a solution. Perhaps this can't be done as a built-in
> syntax but what about a more complex solution. I'm a noob (and with
> the numbers of features of zsh, I'm gonna be a noob forever), and cant
> find a small, beautiful, zsh-is-so-cool-ish solution for it and dont
> know how to effective implement it e.g., without using python, I like
> some zsh only solution.

Unfortunately, no.

I am running into this need very often - be it a small script to convert a
bunch of media files, or a script to download a bunch of web pages, and so
on, and always need to come up with some ugly half-working solution.
I am still really surprised that no shell (that I know) comes with a
convenient built-in syntax to do such loops.

At one point I decided to go ahead an modify zsh myself. After some
deliberation with myself, I came up with the following syntax:

	for i in 1 2 3
	parallel 2 do
		echo $i
		sleep 3
	done

I.e., one adds the keyword "parallel" and the number of concurrent processes
just before the "do/done" block. This syntax makes it easy to parrelize
all kinds of for loops (C-like, csh-like, bourne-shell-like, etc.) in
one syntax.

What this should have done is to run the do/done block in the background
(like with a &), and additionally block while 2 of these are already running,
waiting until one of them stopped (we know when one stops because the shell
has a SIGCHLD interrupt handler).

Unfortunately, I found understanding the zsh parser much harder than I
had originally anticipated. I managed to add the "parallel" syntax, but
was not able (in the several hours I investing in trying to understand)
how to generate the correct "instructions" (of the zsh virtual machine)
to put the do/done block in the background, for example. All the tricks
I tried with WCB_LIST, WCB_SUBLIST, WCB_PIPE, set_list_code, WCB_END and
other strange things I tried, stopped short of actually working.
And even if I had managed to pull that off, I still had some more missing
pieces, like keeping a list of process ids of these backgrounded processes,
recognizing (in the interrupt handler) when they're gone, and waiting until
one of them is gone.

> How people solved this problem any home-made solution for this, any tip?

By the way, in my original thread I mentioned a post from 10 years ago (!)
which suggested an eleborate trick to do what I was after:

        http://www.zsh.org/mla/users/1999/msg00644.html

Have you tried this?


-- 
Nadav Har'El                        |       Sunday, Nov 22 2009, 6 Kislev 5770
nyh@xxxxxxxxxxxxxxxxxxx             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Seen on the back of a dump truck:
http://nadav.harel.org.il           |<---PASSING SIDE . . . . . SUICIDE--->