Scripting qsub

Now that we know how to monitor jobs, let’s return to qsub and discuss how to submit multiple jobs simultaneous. One way is to script a loop that calls qsub many times (unfortunately, aliases aren’t available in scripts, so we have to specify all the options):

File:
batch-sim 
#!/bin/bash
for ((i = 1; i <= 10; i++))
do
  qsub -cwd -V -e ~/err -o ~/out -q TUKEY sim $i
done

Now we can run

> 
./batch-sim

which will make 10 requests to the batch scheduler, all of which should run and finish more or less simultaneously (unless there are less than 10 available processors on TUKEY). After submitting this command, try monitoring the job with qstat.

qsub arrays

Alternatively, qsub itself supports basic looping at the command line with what it calls array jobs. Essentially, qsub itself sets up a loop like the one above; you can access the loop index through the environment variable $SGE_TASK_ID. So, let’s rewrite the sim executable:

File:
sim 
#!/bin/bash
R CMD BATCH --no-save --no-restore "--args $SGE_TASK_ID" sim.R .sim.Rout

Now, we can run

> 
qsub -t 1-15 sim

This will accomplish the same thing as the scripted loop above, but without the need to write a batch-sim file.