Re: Multi-core loops

["Nadav Har'El" <nyh@xxxxxxxxxxxxxxxxxxx>]

On Sun, Jun 28, 2009, Nadav Har'El wrote about "Multi-core loops":
>...
> 	for i in ...
> 	do
> 		dosomething $i
> 	done
> 	
> But when "dosomething" is CPU intensive, this is *not* what you'd want to
> do on a multi-core (multi CPU) machine, which have more-or-less become
> standard nowadays...
> Such a loop would only use one of the CPUs, and leave the other(s) unused.
> Instead, you'll want to keep all CPUs busy all the time, running M (=number
> of CPUs) processes at the same time.
>...

Sorry for replying to my own emails, but I just remembered another point.
The parallel loop I propose to add is not only relevant to SMP machines
and CPU-intensive tasks.

It is also important, sometimes even more important, for some types of
non-CPU intensive tasks. For example, consider a loop like this for
fetching the content of a list of URLs:

	cat urllist | while read url
	do
		wget $url
	done

The computer will be idle most of the time, as wget will most of the time
just wait for responses from the network. If I could easily tell the loop
to run 10 wgets at a time, this would (in most cases) boost the performance
of this loop almost 10-fold! And this is true even if you have just one CPU.

Searching the web, I found that Chapter 14 "Throttling Parallel Processes"
of Ron Peters' "Shell Script Pearls" is about a similar use case, and his
solution is a very eleborate one using a variant of the coprocess-based
solution in the thread I mentioned earlier from the zsh list 10 years ago.
	
So I think there is definitely a need for such a parallel loop feature.

-- 
Nadav Har'El                        |       Sunday, Jun 28 2009, 6 Tammuz 5769
nyh@xxxxxxxxxxxxxxxxxxx             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Creativity consists of coming up with
http://nadav.harel.org.il           |many ideas, not just that one great idea.