For many MATLAB applications, running them in the interactive MATLAB window is the natural and preferred method. However, many other applications, such as running long duration production runs that do not require interaction, are best run in the background mode, commonly referred to as batch jobs. Batch jobs are typically managed by a batch scheduler. Users submit batch jobs via a job submission command and the rest is handled by the batch scheduler and the operating system. The general procedure for batch submission and handling is described in the Shared Computing Center’s Running Jobs page. Please visit that page before delving into the specifics of MATLAB batch processing.
A brief batch system overview
For a complete description of general batch system usage rules, resources, and examples, please visit the running-jobs page. Summarized below are information and procedures pertinent to running MATLAB batch jobs.
- The batch system is Open Grid Scheduler (OGS), the open source version of Sun Grid Engine.
- Batch jobs are submitted to the OGS via a command called
qsub. You can find out more about it with
scc1% man qsub
- This page uses the word processor to denote what computer hardware vendors call a processor core. The OGS
qsubmanpage also use the words job slot or simply slot to refer to the same concept.
- A user can submit as many jobs as needed. However, no more than 256 processors requested by the same user can be in run state at the same time. For users that run MATLAB jobs frequently (especially many at a time), we recommend that you compile your application into a standalone executable. This executable will run directly on the host system (such as the SCC) rather than in the MATLAB environment and hence no MATLAB license will be required. This will prevent your jobs from getting aborted in the event that MATLAB licenses are not available.
- The default wall clock limit is 12 hours. Specify a different limit with
qsub -l h_rt=HH:MM:SS.
- Technical Summary lists available SCC compute nodes with details.
Batch jobs are submitted with the qsub command. The general form of the command is:
scc1% qsub [qsub options] command [arg1 ...]
command is a user supplied shell script which prescribes the operational procedures as well as
Job running procedures vary depending on the type of applications. Instructions for several important types of serial and multiprocessing jobs are demonstrated below through different
command batch scripts.
Serial Jobs | Embarrasingly Parallel Jobs | Parallel Jobs | Jobs Dependency Control
- Running a serial MATLAB batch job with a script.
- Here is a sample batch script,
#!/bin/csh matlab -nodisplay -singleCompThread -r "n=4, myfct(n); exit"
The space between double quotes functions essentially like a MATLAB command window to run any supported MATLAB commands: define a variable (
n=4); run built-in MATLAB utilities (
rand, exit) or user-defined function or script m-file (e.g.,
myfct.m; don’t include
.m! ). The MATLAB
exitcommand is recommended to properly quit MATLAB and ends the batch job. As this is a single-processor job, the
-singleCompThreadturns off implicit parallelism (i.e., using multithreads) to prevent the system from killing jobs that use more resources than requested.
Advice: Optionally, give
mbatchexecute attribute to enable it to run from the window session. This is useful for testing
mbatchin your window session before submitting it via
scc1% chmod +x mbatch scc1% mbatch
- Submit a batch job
scc1% qsub mbatch
qstatto query the status of your jobs
scc1% qstat -u kadin job-ID prior name user state submit/start at queue . . . ---------------------------------------------------------------------- 477578 0.00000 mbatch kadin qw 05/01/2013 08:50:06
In the above,
qwindicates that the job is waiting in queue. A running job would have a state of
- To kill a job already in the queue (running or waiting)
scc1% qdel 477578
- Two output files will be generated in connection with the job; all errors are reported in
mbatch.e477578while output goes to
mbatch.o477578(for MATLAB jobs this includes the MATLAB splash screen and anything that goes to the command window). Please read running-jobs page to find out how to request notification when the batch job is completed.
- Many MATLAB vector commands, especially level-3 (O(n3)) linear algebra operations (like matrix-matrix multiply), can take advantage of multithreading if the amount of computation is significant. If you are sure that your MATLAB code can take advantage of implicit parallel processing, remove the
mbatch, then submit the job to a multiprocessor queue and request an entire node (See running-jobs for details on multiprocessor queues.)
scc1% qsub -pe omp 16 mbatch
- The batch scheduler has built-in default behaviors, like 12-hour wall time limit. If your jobs always require more time or if your jobs’ execution order need control, see Advanced Topics for instructions.
- Here is a sample batch script,
- Bundling serial jobs to run concurrently
This is commonly know as embarrassingly parallel jobs. If you need to run many jobs that are mutually independent, this method is very flexible and convenient, not to mention that no knowledge of the MATLAB Parallel Computing Toolbox is needed. More details . . .
- Single MATLAB task with multiple processors
This is a Parallel Computing Toolbox job. More details . . .
- Running multiple batch jobs in specific order
Read this page to learn how to control the order of execution of multiple batch jobs.