This page demonstrates procedures for running MATLAB applications through the SCC’s batch scheduler. For MATLAB operations such as code development, GUI and other graphical rendering, interactive MATLAB window is the natural and preferred mode of operation. Other applications, such as running long duration production runs that do not require interaction, are best run in the background mode, commonly known as batch. On the SCC, batch jobs are managed by the Open Grid Scheduler (OGS). Users submit batch jobs via a job submission command (
qsub) and the rest is handled by the batch scheduler and the operating system. MATLAB batch job submission and handling generally follow the guidelines detailed in the Shared Computing Cluster’s Running Jobs page. Users running large number of MATLAB jobs may require additional steps for efficient and robust batch operations. These will be elaborated and demonstrated below where necessary.
Batch System Usages & Policies
- Batch jobs are submitted to the batch scheduler via
scc1$ qsub [qsub options] user-script [arg1 ...]
user-scriptis a user supplied shell script that dictates operations to perform while
qsub optionslet you specify supported options. Many
qsuboptions can be included in the user script to serve as default. If a qsub option appears both as qsub command-line input and in user script, the former overrides the latter.
- This page uses the words processor, core, thread and slot to interchangeably denote what computer hardware vendors call a processor core.
- A user can submit as many jobs as needed. However, no more than 256 processors requested by the same user can be in run state at any time.
- The system default wall clock limit is 12 hours. Specify a different limit with
scc1$ qsub -l h_rt=HH:MM:SS . . .
- Serial MATLAB batch jobs can run for up to 720 wall clock hours (30 days)
scc1$ qsub -l h_rt=720:00:00 . . .
- Multicore MATLAB batch jobs using the “omp” parallel environment can run for 720 hours
scc1$ qsub -pe omp 16 -l h_rt=720:00:00 . . .
- At present, there is no provision for running MATLAB jobs exceeding the maximum workers (or cores) permitted by the Parallel Computing Toolbox and hence the shared-memory
-pe ompis the only Parallel Environment selection. For the current system default R2013a release, this limit is 12 cores. When R2014a and R2014b are available for general usage, the limit will be 16 cores.
With R2013a release, even if you request 16 cores with -pe omp 16, the MATLAB Parallel Computing Toolbox would only let you use 12 cores, leaving the remaining 4 cores unused.
- Technical Summary lists available SCC compute nodes with details.
Essential Batch Commands
qsubto submit batch jobs. For example
scc1$ qsub ./mbatch
mbatchis a basic batch script to run MATLAB using 1 processor
#!/bin/csh matlab -nodisplay -singleCompThread -r "n=4, rand(n), exit"
Represented between the pair of double quotes above is, effectively, a MATLAB command window to run supported MATLAB commands: define a variable (
n=4); run built-in MATLAB utilities (
rand, exit); or user m-files, say,
myfct(omit the .m suffix). The MATLAB
exitcommand ensures proper ending of the MATLAB session and batch job. By default, this is a single-processor job. The
-singleCompThreadprevents MATLAB from invoking multithreading (i.e., using multicore) automatically which would trigger system abort for overusage of system resources. The
-nodisplayruntime switch suppresses rendering for batch jobs. Graphics may be rendered and saved as an image file with the MATLAB
mbatchscript provides a very simple script for basic, single processor MATLAB batch jobs. Different MATLAB applications may require appropriate changes to the basic script. These various types are demonstrated below in Types of MATLAB Batch Jobs.
While not required as a batch script, enabling
mbatchas an executable extends its functionality as a command — handy for error-checking before use in batch processing.
scc1$ chmod +x mbatch scc1$ ./mbatch
qstatto query batch queue status. Add
-uoption to list specific jobs
scc1$ qstat -u myID job-ID prior name user state submit/start at queue slots ja-task-ID ------------------------------------------------------------------------------------------------------------- 6860722 0.96195 myJobs myID r 03/23/2015 10:20:10 email@example.com 1 6860723 0.00000 myJobs myID qw 03/23/2015 10:20:11 1 6860724 0.00000 myJobs myID Eqw 03/23/2015 10:20:20 1
In the above,
qwindicates job is waiting while
rmeans job is running. A state of
Eqwindicates job in error. Use
qstat -j 6860724for explanations.
- To kill a job in the queue
scc1$ qdel 6860723
- Output goes to
myJobs.o6860722(this includes the MATLAB splash screen and anything that goes to the command window). More details on
qsuboptions, such as output control, are available in running-jobs page.
- The batch scheduler has built-in system default behaviors, like 12-hour wall time limit. You can define your own qsub default behaviors so that you won’t have to specify them. If you wish to control the execution order of your batch jobs, see Advanced Batch System Usage.
Depending on the applications, MATLAB batch job running procedures generally fall into one of the following categories. Sample batch scripts and utilities discussed in this page are available for download or copy (for SCC users).
scc1$ cp -r /project/scv/examples/matlab/batch/ your-SCC-dir-path
Running single-core (serial) MATLAB Batch Jobs
run_matlab_job, an enhanced version of
mbatch, you can specify arbitrary problem size at runtime.
scc1$ qsub ./run_matlab_job 5 # compute rand(5)
#!/bin/csh matlab -nodisplay -singleCompThread -r "rand($1), exit"
qsubruns, system shell (
run_matlab_jobexpects one runtime input (random array size
$1(additional $2, $3, … if required).
- Shell substitutes variable $1 with 5, runs
matlab . . -r "rand(5),exit"
To adapt it for your own app, replace
rand($1)with your m-file (omit
If you are not submitting a batch job from the directory where your app resides, you may need to use
addpath, for example, to let MATLAB knows where to find it.
See Running Multiple Batch Jobs on ways to run a group of similar jobs.
Running Multi-core MATLAB Parallel Computing Toolbox Batch Jobs
runBatchJob.mbelow to keep this PCT-related I/O traffic within the compute node to forstall undesirable inter-node data communications.
scc1$ qsub ./run_matlab_pct_job # 4 cores; n=100 scc1$ qsub -pe omp 8 ./run_matlab_pct_job # 8 cores; n=100 scc1$ qsub -pe omp 8 -v n=200 ./run_matlab_pct_job # 8 cores; n=200
#!/bin/csh #$ -pe omp 4 # set default value for n; override with qsub -v at runtime #$ -v n=100 # Additional qsub options here . . . matlab -nodisplay -r "runBatchJob($n, $NSLOTS); exit"
In general, a PCT job is expected to run on multicores. The example batch script presets it to 4 cores (#$ -pe omp 4), which can be overridden at runtime. See Custom qsub settings with .sge_request for details on override rules.
The double-quoted command region in
matlab ... -r " ... "(first explained in the Serial Batch section) is preprocessed by the system shell in
run_matlab_pct_jobto replace valid environment variables, like $n, with their respective values before being passed to the matlab command.
- n (defaulted to 100) is used to compute s = 1 + 2 + … + n
- $NSLOTS is set to 4 per qsub option statement
#$ -pe omp 4
(In above example, $NSLOTS=8 if runtime override exercised.)
It is prudent to “inherit” a pre-defined qsub option value, per $NSLOTS. Defining it explicitly, e.g., runBatchJob(3, 4), defeats the purpose of runtime override. It may also lead to inadvertent inconsistencies which may lead to job abortion if system resource overuse occurs.
function runBatchJob(n, nslots) % computes sum s=1+2+3+...+n=n(n+1)/2 with nslots cores % redirects ~/.matlab PCT temp files to TMPDIR on the compute % node to avoid inter-node (compute node <--> login node) I/O myCluster = parcluster('local') % cores on compute node to be "local" if getenv('ENVIRONMENT') % true if this is a batch job myCluster.JobStorageLocation = getenv('TMPDIR') % points to TMPDIR end % REPLACE BELOW EXAMPLE WITH YOUR APP (either scripts or functions) %parpool(myCluster, nslots) % for MATLAB R2014a or newer matlabpool(myCluster, nslots) % for MATLAB R2013a or older s = 0; parfor i=1:n s = s + i; % compute s = 1 + 2 + ... + n = n(n+1)/2 end fprintf(1,'Computed arithmetic sequence sum s = %d', s); fprintf(1,' (correct answer: %d)\n\n\n', n*(n+1)/2); matlabpool close % MATLAB 2013a or older %delete(gcp) % MATLAB 2014a or newer end
See Running Multiple Batch Jobs on simple way to run a group of jobs.
Running Pre-compiled MATLAB Standalone Jobs
qsub‘s Array Job option (
qsub -t) makes it easy to run a series of jobs with a single launch of
qsub. For example,
scc1$ qsub -t 100-300:200 ./run_standalone_job
Running Implicitly Parallel MATLAB Batch Jobs
mbatchand submit the job to a multiprocessor queue. There are two situations associated with implicit parallel computation in regards to batch processing: using a whole node (SCC hardware options are 12 and 16 cores) versus using a partial node.
- Sample batch script for using a whole node,
#!/bin/csh matlab -nodisplay -r "A=rand(1e4); I=A/A; exit"
scc1$ qsub -pe omp 16 ./wnbatch
In this example, 16 cores are assigned to the job but MATLAB R2013a only allow up to 12 cores. Starting with MATLAB R2014a (not yet available on the SCC), all 16 cores will be permitted.
- Sample batch script for using a partial node,
#!/bin/csh matlab -nodisplay -r "maxNumCompThreads($NSLOTS); A=rand(1e4); I=A/A; exit"
scc1$ qsub -pe omp 8 ./pnbatch
qsubcommand line option -pe omp 8 passes the core count to the batch scheduler’s $NSLOTS environment variable at runtime.
The use of
maxNumCompThreadscauses a warning of its future deprecation. As of MATLAB R2014b, this function is still supported.
Note the absence of MATLAB option -singleCompThread in
Running Multiple Batch Jobs With qsub Array Job Option
qsub -t(aka Array Job) option is ideally suited for parametric studies. Note that if the system load permits, jobs under a parametric study run concurrently — and effectively as Embarrassingly Parallel jobs — they practically scale linearly. The
qsub -toption essentially has the effect of PCT’s
parfor. The qsub manpage states that “The option argument to -t specifies the number of array job tasks and the index number which will be associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID.” The relationship between SGE_TASK_ID and the -t input argument is demonstrated below, along with examples
% qsub -t SGE_TASK_FIRST — SGE_TASK_LAST : SGE_TASK_STEP . . . % qsub generates SGE_TASK_ID for each job with above indices % Simulate SGE_TASK_ID with MATLAB colon syntax -- NOT REQUIRED SGE_TASK_ID=SGE_TASK_FIRST:SGE_TASK_STEP:SGE_TASK_LAST; = [100,300,500]; % qsub -t 100-500:200 (run 3 jobs) = 100; % qsub -t 100 (run 1 job) = [3,4,5]; % qsub -t 3-5 (default step size 1)
qsub -t 100-300will spawn 201 batch jobs because the default step size is 1 ! Anticipatorily, name your array job
scc1$ qsub -N myJobs -t 100-300:200 . . .
In the event that an array job was submitted by mistake, simply delete all tasks with
scc1$ qdel -u yourUSERID "myJobs*"
Single-Core Parametric Studies
To perform single-core parametric study, one could use
run_matlab_job multiple times, each with a different value of
qsub -v n=N ... However, it is more convenient to run the jobs as an array, with which the environment variable $SGE_TASK_ID is used directly (or indirectly, like
n=fct($SGE_TASK_ID)) as the random matrix order
scc1$ qsub -t 3-7:2 ./run_matlab_aj
#!/bin/csh #$ -v alpha=1 #$ -v beta=2 # use env var SGE_TASK_ID as random matrix size n matlab -nodisplay -singleCompThread \ -r "myRand($SGE_TASK_ID, $alpha, $beta), exit"
As previously explained in
run_matlab_job, the double quoted segment in
run_matlab_ajis first processed by the system shell to replace all specified environment variables with their respective values before passing control on to MATLAB. The above example runs 3 jobs with $SGE_TASK_ID passed to each as 3, 5, and 7, respectively as the direct substitute for
n, the random square matrix order.
function myRand(n, alpha, beta) % This is a companion to run_matlab_aj batch script to demonstrate % qsub -t option. It computes a random matrix, then save output of % task to a file with name indicative of task A = rand(n); % computes random matrix filnam=['output_' num2str(n)]; % name of file to be saved save(filnam, 'A','alpha','beta'); % saves A to a mat file
n(= $SGE_TASK_ID) is also used to define I/O file name for individual tasks to prevent all tasks writing to the same output file.
Multi-Core (PCT) Parametric Studies
scc1$ qsub -t 100 -pe omp 8 ./run_matlab_pct_aj # 8 cores; n=100 scc1$ qsub -t 100-300:200 ./run_matlab_pct_aj # 4 cores; n=100, 300
#!/bin/csh #$ -pe omp 4 # Additional qsub options here . . . matlab -nodisplay -r "runBatchJob($SGE_TASK_ID, $NSLOTS); exit"
Using as many cores as available in a node has several advantages:
- More cores may help to speedup computation;
- Being the sole user on the node guarantees that all memory available to node will not be shared with another job;
- Having multiple PCT jobs running on the same node may trigger crash.
Parametric Studies Requiring Multiple Parameters
Here are a few ways to run parametric studies with multiple variables . . .
N - 1variables fixed (e.g.,
alpha, beta) and run multiple jobs on remaining
Nth variable with Array Job
scc1$ qsub -v alpha=1 -v beta=2 -t 10-15 ./run_matlab_aj scc1$ qsub -v alpha=3 -v beta=4 -t 10-15 ./run_matlab_aj
- The above can be further automated, e.g., with this runjobs script
#!/bin/csh # runjobs shell script foreach a (`seq 1.1 0.3 3.2`) foreach b (`seq 1 1 3`) qsub -v alpha=$a -v beta=$b -t 10-15 run_matlab_aj end end
The above script uses the Unix sequence command
seqto generate integer and floating-point sequences to feed the for-loop indices
b. Input to
seqhas this format:
first:step:last. Note that the symbol ` is a left-single-quote. In this case
a = [1.1, 1.4, 1.7, 2.0, 2.3, 2.6, 2.9, 3.2]; % seq 1.1 0.3 3.2 b = [1, 2, 3]; % seq 1 1 3
Don’t forget to give
scc1$ chmod +x ./runjobs
Then, executing runjobs yields
scc1$ ./runjobs Your job-array 6826755.10-15:1 ("runjobs") has been submitted . . . . . . . . . .
- Alternatively, the MATLAB
ind2subutility may be used to map Array Job’s linear indexing to N-Dimensional indexing. For example, if you want to generate a 3×4 array of 2-D indexing, you could submit an array job:
scc1$ qsub -t 1-12 . . .
This launches 12 jobs with their respective
$SGE_TASK_ID = 1, 2, 3, . . ., 12. Next, in
myApp.m. . .
% with SGE_TASK_ID (say, 7) passed into your myApp.m ... [i, j] = ind2sub([3 4], SGE_TASK_ID) % returns [row, col] i = 1 j = 3 alpha = Alpha(i); % Alpha is independent of j beta = Beta(j); % Beta is independent of i