Skip to content

Instantly share code, notes, and snippets.

@musically-ut

musically-ut/poor_man_parallel.sh

Last active April 1, 2019 10:11
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save musically-ut/9334018d9a66c44ff0d271c602a6ad1c to your computer and use it in GitHub Desktop.
Save musically-ut/9334018d9a66c44ff0d271c602a6ad1c to your computer and use it in GitHub Desktop.
Download ZIP
Poor man's parallel execution on servers
Raw
poor_man_parallel.sh
#!/usr/bin/env bash
set -e
for N in {1..5}
do
for M in {0.1,1,2,5,10};
do
# Replace ./my_prog.py with your program to run it in parallel
./my_prog.py -N $N -M $M > output/stdout_N${N}_M${M}.txt &
echo $!
done
done
## Run as:
## ./pool_man_parallel.sh > pids
## Then kill recalcitrant processes from `pids` file, if needed.
@harrymvr
Copy link

Nice one! As an alternative, you could kill all the processes by killing the pid of the parent, e.g. pkill -P <parent_pid>.

Also, if you are not sure if you have any bugs in my_prog.py, and you want the script to die and stop spawning new processes if an exception happens, you can use set -e in the beginning.

@musically-ut
Copy link
Author

@harrymvr Thanks for the set -e idea, I've included it in the script. I didn't know about pkill -P trick, but won't it kill the shell itself?

@harrymvr
Copy link

@musically-ut pkill -P does not kill the TERM, but I think that it killed the forked processes. Let me know, because I might need to change that in my scripts too 😄

@musically-ut
Copy link
Author

@harrymvr The PPID of the processes put in background seems to be 1, i.e. the init process:

I am not too keen on running pkill -P 1. <_<

@harrymvr
Copy link

hmmm interesting. Yeah, I would pass on that as well 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment