Threaded Parallel Environment

Historically the parallel environments on the Engineering Grid (Eng-Grid) and SCC used different terms for the multi-threaded parallel environment. The SCC used omp for what Eng-Grid called threaded. The actual job behavior was similar, and now you can use both terms on Eng-Grid.

If you want to see the grid’s list of parallel environments, run qconf -spl. To see the settings for a given one, run (for example) qconf -sp threaded.

Question:

I have a couple tasks that use pthreads and or openmp - some of the tasks
use all 8 cpu on a node - what is the best way to allocate slots for these
kind of tasks - I could write an mpi version of these but it is far simpler
to use openmp/pthreads?

Answer:

I have created a parallel environment called "threaded" for this purpose.  Submit using the switch
"-pe threaded NUMTHREADS"
where NUMTHREADS is an integer specifying the number of threads your job intends to use on any given node.

This will eat the number of slots on any given node equivalent to NUMTHREADS.  So for example:

enggrid1:/mnt/nokrb/kamalic$ qstat -f|grep budge.q
budge.q@budge01.bu.edu         BIP   0/0/8          0.07     lx26-amd64
budge.q@budge02.bu.edu         BIP   0/1/8          0.01     lx26-amd64

enggrid1:/mnt/nokrb/kamalic$ qsub -cwd -q budge.q -pe threaded 8 /mnt/nokrb/sge/examples/jobs/sleeper.sh
Your job 2598980 ("Sleeper") has been submitted

enggrid1:/mnt/nokrb/kamalic$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
2598980 0.51674 Sleeper    kamalic      r     01/18/2012 17:53:05 budge.q@budge01.bu.edu             8

enggrid1:/mnt/nokrb/kamalic$ qstat -f|grep budge.q
budge.q@budge01.bu.edu         BIP   0/8/8          0.06     lx26-amd64
budge.q@budge02.bu.edu         BIP   0/1/8          0.01     lx26-amd64

If I submit another one to budge.q now, it will stay in "qw" until either my 8-slot job on budge01 finishes, or that one-slot job on budge02 finishes.  Basically, ONE of the nodes in budge.q has 8 free slots on it.  It will not run on budge02 immediately because budge02 currently only has 7 free slots on it.

You can submit this way with "-q budge.q,bungee.q,big.q" -- all of these queues have 8 cores per node.

If you only wish to use NUMTHREADS=2, then bme.q will run it as well.  ece.q is curently only configured for 1 slot per node, but I'm tempted to change that back to 2 slots per node so then your jobs could run there too.  (Certainly, if you comma-separate all of the queues you're interested in with "-q budge.q,bungee.q,big.q,ece.q,bme.q" then if you ask for more slots than a particular queue's nodes are even configured for, of course it won't run on those nodes.)