Re: Multi-core loops

[Mikael Magnusson <mikachu@xxxxxxxxx>]

2009/11/22 Nadav Har'El <nyh@xxxxxxxxxxxxxxxxxxx>:
> On Sun, Nov 22, 2009, Kazuo Teramoto wrote about "Re: Multi-core loops":
>> On Sun, Jun 28, 2009 at 8:31 AM, Nadav Har'El <nyh@xxxxxxxxxxxxxxxxxxx> wrote:
>> > Zsh, like all shells, lets you easily do something many times in a loop.
>>...
>> > But when "dosomething" is CPU intensive, this is *not* what you'd want to
>> > do on a multi-core (multi CPU) machine, which have more-or-less become
>> > standard nowadays...
>> > Such a loop would only use one of the CPUs, and leave the other(s) unused.
>> > Instead, you'll want to keep all CPUs busy all the time, running M (=number
>> > of CPUs) processes at the same time.
>>
>> Any update on this?
>>
>> I'm searching for a solution. Perhaps this can't be done as a built-in
>> syntax but what about a more complex solution. I'm a noob (and with
>> the numbers of features of zsh, I'm gonna be a noob forever), and cant
>> find a small, beautiful, zsh-is-so-cool-ish solution for it and dont
>> know how to effective implement it e.g., without using python, I like
>> some zsh only solution.
>
> Unfortunately, no.
>
> I am running into this need very often - be it a small script to convert a
> bunch of media files, or a script to download a bunch of web pages, and so
> on, and always need to come up with some ugly half-working solution.
> I am still really surprised that no shell (that I know) comes with a
> convenient built-in syntax to do such loops.
>
> At one point I decided to go ahead an modify zsh myself. After some
> deliberation with myself, I came up with the following syntax:
>
>        for i in 1 2 3
>        parallel 2 do
>                echo $i
>                sleep 3
>        done
>
> I.e., one adds the keyword "parallel" and the number of concurrent processes
> just before the "do/done" block. This syntax makes it easy to parrelize
> all kinds of for loops (C-like, csh-like, bourne-shell-like, etc.) in
> one syntax.
>
> What this should have done is to run the do/done block in the background
> (like with a &), and additionally block while 2 of these are already running,
> waiting until one of them stopped (we know when one stops because the shell
> has a SIGCHLD interrupt handler).
>
> Unfortunately, I found understanding the zsh parser much harder than I
> had originally anticipated. I managed to add the "parallel" syntax, but
> was not able (in the several hours I investing in trying to understand)
> how to generate the correct "instructions" (of the zsh virtual machine)
> to put the do/done block in the background, for example. All the tricks
> I tried with WCB_LIST, WCB_SUBLIST, WCB_PIPE, set_list_code, WCB_END and
> other strange things I tried, stopped short of actually working.
> And even if I had managed to pull that off, I still had some more missing
> pieces, like keeping a list of process ids of these backgrounded processes,
> recognizing (in the interrupt handler) when they're gone, and waiting until
> one of them is gone.
>
>> How people solved this problem any home-made solution for this, any tip?
>
> By the way, in my original thread I mentioned a post from 10 years ago (!)
> which suggested an eleborate trick to do what I was after:
>
>        http://www.zsh.org/mla/users/1999/msg00644.html
>
> Have you tried this?

I came across this script a while ago, it's written in bash and I
haven't tried it, but it claims to run stuff in parallel :). Maybe it
can be zsh-ified to get rid of the warning about commands probably not
working with spaces.
http://sitaramc.googlepages.com/queue.sh

-- 
Mikael Magnusson