| Perhaps I need to think about this another way and that is via “locking”. From a 10,000 foot view, I could map each target to a file (path) and do the old fashion practice of putting the PID in the file and do it atomically and with the ability to detect that the owner of the lock might have died, etc etc etc. Now just spin off N tasks each with a complete list of targets. The basic flow would be for each item in the list of targets, see if it is done. If it isn’t done, see if it is locked. If it isn’t locked, set the lock (atomically) and then do the processing to create the target. Now, if I want 4 processes running, I can just start four jobs in the back ground.
|