{"id":66367,"date":"2013-05-15T12:10:10","date_gmt":"2013-05-15T16:10:10","guid":{"rendered":"http:\/\/www.bu.edu\/tech\/?page_id=66367"},"modified":"2022-06-07T10:06:29","modified_gmt":"2022-06-07T14:06:29","slug":"matlab-batch","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/tech\/support\/research\/software-and-programming\/common-languages\/matlab\/matlab-batch\/","title":{"rendered":"Running MATLAB Batch Jobs on the SCC"},"content":{"rendered":"<p>This page demonstrates procedures for running MATLAB applications through the SCC&#8217;s batch scheduler. For MATLAB operations such as code development, GUI, and other graphical rendering, an interactive MATLAB window is the natural and preferred mode of operation. Other applications, such as running long duration production runs that do not require interaction, are best run in the background mode, commonly known as <b>batch<\/b>. On the SCC, batch jobs are managed by the <a href=\"http:\/\/gridscheduler.sourceforge.net\/\">Open Grid Scheduler (OGS)<\/a>. Users submit batch jobs via a job submission command (<code>qsub<\/code>) and the rest is handled by the batch scheduler and the operating system. MATLAB batch job submission and handling generally follow the guidelines detailed in the Shared Computing Cluster&#8217;s <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/running-jobs\/\">Running Jobs<\/a> page. Users running a large number of MATLAB jobs may require additional steps for efficient and robust batch operations. These will be elaborated on and demonstrated below where necessary.<\/p>\n<h4>Batch Basics<\/h4>\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Batch System Usages &amp; Policies<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"><\/p>\n<ul>\n<li>Batch jobs are submitted to the batch scheduler via <code><span class=\"command\"><a href=\"http:\/\/scv.bu.edu\/cgi-bin\/perl\/manscript\/UNIX\/qsub\/1\">qsub<\/a><\/span><\/code>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\">[qsub options] user-script [arg1 ...]<\/span><\/code><\/pre>\n<p>Above, <code><span class=\"placeholder\">user-script<\/span><\/code> is a user supplied shell script that dictates operations to perform while <code><span class=\"placeholder\">qsub options<\/span><\/code> let you specify supported options. There are many <code>qsub<\/code> options that can be included in the user script. If a qsub option appears both as <code><span class=\"command\">qsub<\/span><\/code> command-line input and in the user script, the former overrides the latter.<\/li>\n<li>This page uses the words <strong>processor, core, thread<\/strong> and <strong>slot<\/strong> to interchangeably denote what computer hardware vendors call a <strong>processor core.<\/strong><\/li>\n<li>A user can submit as many jobs as needed.<\/li>\n<li><b>The system default wall clock limit is 12 hours.<\/b> Specify a different limit with\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$ <\/span><span class=\"command\">qsub -l h_rt=<\/span><span class=\"placeholder\">HH:MM:SS . . .<\/span><\/code><\/pre>\n<\/li>\n<li>Serial MATLAB batch jobs can run for up to 720 wall clock hours (30 days)\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$ <\/span><span class=\"command\">qsub -l h_rt=<\/span><span class=\"placeholder\">720:00:00 . . .<\/span><\/code><\/pre>\n<\/li>\n<li>Multicore MATLAB batch jobs using the &#8220;omp&#8221; parallel environment can run for up to 720 hours. Take 28 cores for example,\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$ <\/span><span class=\"command\">qsub -pe omp 28<\/span><span class=\"placeholder\"> -l h_rt=720:00:00 . . .<\/span><\/code><\/pre>\n<\/li>\n<li>Note that the <code>-pe omp 28<\/code> only means 28 CPU cores are assigned to the job. Users are responsible to make their Matlab programs really run on these cores. For example, using <code>-maxNumCompThreads(28)<\/code> for implicit parallelism or <code>parpool(28)<\/code> for explicit parallelism. For more details about the &#8220;omp&#8221; parallel environment, please refer to <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/running-jobs\/parallel-batch\/#openmp\">this page<\/a>.<\/li>\n<li>The maximum number of threads is 32 in MATLAB 2014a or newer.<\/li>\n<li>Multi-node parallel computing in MATLAB is not supported on the SCC.<\/li>\n<li>The <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/computing-resources\/tech-summary\/#SCC\">Technical Summary<\/a> lists available SCC compute nodes with details on cores, memory, scratch disk, and more.<\/li>\n<\/ul>\n<p><\/div>\n<\/div>\n\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Essential Batch Commands<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"><a name=\"BATCHCOMMANDS\" id=\"BATCHCOMMANDS\"><\/a><\/p>\n<ol>\n<li style=\"list-style-type: lower-alpha;\">Use <code><span class=\"command\">qsub<\/span><\/code> to submit batch jobs. For example:\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\">.\/mbatch<\/span><\/code><\/pre>\n<p>where <code><span class=\"placeholder\">mbatch<\/span><\/code> is a basic batch script to run MATLAB using 1 processor:<\/p>\n<pre class=\"code-block\"><code>#!\/bin\/bash -l\r\n<span class=\"command\">module load matlab<\/span>\r\n<span class=\"command\">matlab -nodisplay -singleCompThread <\/span>-r \"<em>n=4, rand(n), exit<\/em>\"<\/code><\/pre>\n<p>Represented between the pair of double quotes above is, effectively, a MATLAB command window to run supported MATLAB commands: define a variable ( <code>n=4<\/code> ); run built-in MATLAB utilities ( <code>rand, exit<\/code> ); or user m-files, say, <code>myfct<\/code> (omit the .m suffix). The MATLAB <code>exit<\/code> command ensures proper ending of the MATLAB session and batch job. By default, this is a single-processor job. The <code>-singleCompThread<\/code> prevents MATLAB from invoking <a href=\"https:\/\/www.bu.edu\/tech\/z-retired-pages\/matlab-pct\/implicit-parallelism\/\">multithreading<\/a> (<i>i.e.,<\/i> using multicore) automatically which would cause the job to be killed by the system for overusage of system resources. The <code>-nodisplay<\/code> runtime switch suppresses rendering for batch jobs. Graphics may be rendered and saved as an image file with the MATLAB <code>print<\/code> utility for viewing subsequently in an interactive window session.<\/p>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p>The <code><span class=\"placeholder\">mbatch<\/span><\/code> script provides a very simple script for basic, single processor MATLAB batch jobs. Different MATLAB applications may require appropriate changes to the basic script. These various types are demonstrated below in <i>Types of MATLAB Batch Jobs<\/i>.<\/p><\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>While not required as a batch script, enabling <code><span class=\"placeholder\">mbatch<\/span><\/code> as an executable extends its functionality as a command \u2014 handy for error-checking before use in batch processing.<\/em><\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">chmod +x<\/span> <span class=\"placeholder\">mbatch<\/span>\r\n<span class=\"prompt\">scc1$<\/span> <span class=\"command\">.\/mbatch<\/span><\/code><\/pre>\n<\/blockquote>\n<\/li>\n<li style=\"list-style-type: lower-alpha;\">Use <code><span class=\"command\">qstat<\/span><\/code> to query batch queue status. Add <code><span class=\"command\">-u<\/span><\/code> option to list jobs for a specific user:\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qstat -u<\/span> <span class=\"placeholder\">myID<\/span>\r\n<span class=\"output\">job-ID  prior   name    user   state submit\/start at     queue                  slots ja-task-ID  \r\n-------------------------------------------------------------------------------------------------------------\r\n6860722 0.96195 myJobs  myID   r     03\/23\/2015 10:20:10 b@scc-ba6.scc.bu.edu     1\r\n6860723 0.00000 myJobs  myID   qw    03\/23\/2015 10:20:11                          1\r\n6860724 0.00000 myJobs  myID   Eqw   03\/23\/2015 10:20:20                          1<\/span><\/code><\/pre>\n<p>In the above, <code>qw<\/code> indicates that the job is waiting while <code>r<\/code> means the job is running. A state of <code>Eqw<\/code> indicates and error with the job. Use <code><span class=\"command\">qstat -j<\/span> <span class=\"placeholder\">6860724<\/span><\/code> to see an explanation for the error with that particular job.<\/li>\n<li style=\"list-style-type: lower-alpha;\">To kill a job in the queue:\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qdel<\/span> <span class=\"placeholder\">6860723<\/span><\/code><\/pre>\n<\/li>\n<li style=\"list-style-type: lower-alpha;\">Output goes to <code><span class=\"placeholder\">myJobs.o6860722<\/span><\/code> (this includes the MATLAB splash screen and anything that goes to the command window). More details on <code><span class=\"command\">qsub<\/span><\/code> options, such as output control, are available on the <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/running-jobs\/\">Running Jobs<\/a> page.<\/li>\n<li style=\"list-style-type: lower-alpha;\">The batch scheduler has built-in system default behaviors, like a 12-hour wall time limit. You can define your own <code><span class=\"command\">qsub<\/span><\/code> default behaviors so that you won&#8217;t have to specify them each time on the command line or in your batch script. To do this or to control the execution order of your batch jobs, see <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/running-jobs\/advanced-batch\/\">Advanced Batch System Usage<\/a>.<\/li>\n<\/ol>\n<p><\/div>\n<\/div>\n\n<h4><a name=\"TYPES\" id=\"TYPES\"><\/a>Types of MATLAB Batch Jobs<\/h4>\n<p>Depending on the applications, MATLAB batch job running procedures generally fall into one of the following categories. Sample batch scripts and utilities discussed in this page are available for <b>download or copy<\/b> (for SCC users).<a name=\"FILES\" id=\"FILES\"><\/a><\/p>\n<pre><code class=\"code-block\"><span class=\"prompt\">scc1$<\/span> <span class=\"command\">cp<\/span> -r <a href=\"http:\/\/scv.bu.edu\/examples\/matlab\/batch\/\" target=\"new\" rel=\"noopener noreferrer\"><em>\/project\/scv\/examples\/matlab\/batch<\/em><\/a>  <span class=\"placeholder\">your-SCC-dir-path<\/span><\/code><\/pre>\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Running single-core (serial) MATLAB Batch Jobs<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"><a name=\"SERIAL\" id=\"SERIAL\"><\/a>With <code><span class=\"placeholder\">run_matlab_job<\/span><\/code>, an enhanced version of <a href=\"#BATCHCOMMANDS\"><code>mbatch<\/code><\/a>, you can specify an arbitrary problem size at runtime.<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\">.\/run_matlab_job 5         # compute rand(5)<\/span><\/code><\/pre>\n<p><a name=\"RUN_MATLAB_JOB\" id=\"RUN_MATLAB_JOB\"><\/a><code><span class=\"placeholder\">run_matlab_job<\/span><\/code><\/p>\n<pre class=\"code-block\"><code>#!\/bin\/bash -l\r\nmodule load matlab\r\nmatlab -nodisplay -singleCompThread -r \"<em>rand($1), exit<\/em>\"<\/code><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>When <code><span class=\"command\">qsub<\/span><\/code> runs, the system shell (<code>bash<\/code>) parses <code>run_matlab_job<\/code> like this:<\/em><\/p>\n<ul>\n<li><code>run_matlab_job<\/code> expects one runtime input (random array size <code>n<\/code>) labelled <code>$1<\/code> (additional $2, $3, &#8230; if required).<\/li>\n<li>Shell substitutes variable $1 with 5, runs <code>matlab . . -r \"rand(5),exit\"<\/code><\/li>\n<\/ul>\n<\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>To adapt it for your own program, replace <code>rand($1)<\/code> with your m-file (omit <code>.m<\/code>).<\/em><\/p><\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>If you are not submitting a batch job from the directory where your program resides, you may need to use <code>addpath<\/code>, for example, to let MATLAB knows where to find it.<\/em><\/p><\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>See <a href=\"#MULTIPLE\">Running Multiple Batch Jobs<\/a> on ways to run a group of similar jobs.<\/em><\/p><\/blockquote>\n<p><\/div>\n<\/div>\n\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Running Multi-core MATLAB Parallel Computing Toolbox Batch Jobs<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"> Here are example commands to submit batch jobs for running multi-core MATLAB PCT programs:<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub <\/span> <span class=\"placeholder\">.\/run_matlab_pct_job                    # 4 cores; n=100<\/span>\r\n<span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -pe omp 8<\/span> <span class=\"placeholder\">.\/run_matlab_pct_job           # 8 cores; n=100<\/span>\r\n<span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -pe omp 8 -v n=200<\/span> <span class=\"placeholder\">.\/run_matlab_pct_job  # 8 cores; n=200<\/span><\/code><\/pre>\n<p><code><span class=\"placeholder\">run_matlab_pct_job<\/span><\/code>:<\/p>\n<pre class=\"code-block\"><span class=\"placeholder\">#!\/bin\/bash -l\r\n#$ -pe omp 4\r\n# set default value for n; override with qsub -v at runtime\r\n#$ -v n=100\r\n# Load the newest version of matlab on SCC\r\nmodule load matlab\r\n# Additional qsub options here . . .\r\nmatlab -nodisplay -r \"runBatchJob($n, $NSLOTS); exit\"<\/span><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>In general, a PCT job is expected to run on multiple cores. The example batch script presets it to 4 cores (#$ -pe omp 4), which can be overridden at runtime. See <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/running-jobs\/advanced-batch\/#sge_request\">Custom qsub settings with .sge_request<\/a> for details on override rules.<\/em><\/p><\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>The double-quoted command region in <code>matlab ... -r \" ... \"<\/code> (first explained in the <a href=\"#SERIAL\">Serial Batch<\/a> section) is preprocessed by the system shell in <code><span class=\"placeholder\">run_matlab_pct_job<\/span><\/code> to replace valid environment variables, like $n, with their respective values before being passed to the <code><span class=\"command\">matlab<\/span><\/code> command.<\/em><\/p>\n<ol>\n<li><b>n<\/b> (defaulted to 100) is used for the computation in the MATLAB program.<\/li>\n<li><b>$NSLOTS<\/b> is set to 4 per qsub option statement <code>#$ -pe omp 4<\/code><br \/>\n(In the above example, $NSLOTS=8 if runtime override exercised.)<\/li>\n<\/ol>\n<p>It is prudent to &#8220;inherit&#8221; a pre-defined <code><span class=\"command\">qsub<\/span><\/code> option value, per $NSLOTS. Defining it explicitly, <i>e.g.,<\/i> runBatchJob(3, 4), defeats the purpose of runtime override. It may also lead to inadvertent inconsistencies which may lead to the job being killed if system resource overuse occurs.<\/p><\/blockquote>\n<p>Each time the PCT is invoked, for internal administrative purposes, MATLAB creates a group of files and folders in the user&#8217;s home directory (<code>~\/.matlab<\/code>). When an interactive MATLAB PCT job is running on one of the login nodes, data flow between the application and <code>~\/.matlab<\/code> is local and it poses no communication issues. A batch job, on the other hand, is dispatched to a compute node at runtime and the aforementioned communication is between the login node and the compute node. Generally, inter-nodes communications are much less efficient than local (<i>i.e.,<\/i> intra-node) communications; frequent occurrences could lead to undue I\/O traffic bottleneck for the system as well as performance degradation of the application.<\/p>\n<div class=\"highlight-yellow\">Please follow the procedure demonstrated in the <code>runBatchJob.m<\/code> below to keep this PCT-related I\/O traffic within the compute node to forstall undesirable inter-node data communications.<\/div>\n<p><a name=\"RUNBATCHJOB\" id=\"RUNBATCHJOB\"><\/a><code><span class=\"placeholder\">runBatchJob.m<\/span><\/code><\/p>\n<pre class=\"code-block\"><span class=\"placeholder\">function runBatchJob(n, nslots)\r\n% computes sum s=1+2+3+...+n=n(n+1)\/2 with nslots cores\r\n% redirects ~\/.matlab PCT temp files to TMPDIR on the compute\r\n% node to avoid inter-node (compute node &lt;--&gt; login node) I\/O\r\n<em>\r\nmyCluster = parcluster('local') % cores on compute node to be \"local\"\r\nif getenv('ENVIRONMENT')    % true if this is a batch job\r\n  myCluster.JobStorageLocation = getenv('TMPDIR')  % points to TMPDIR\r\nend\r\n<\/em>\r\n% REPLACE BELOW EXAMPLE WITH YOUR APP (either scripts or functions)\r\nparpool(myCluster, nslots)    % for MATLAB R2014a or newer\r\n%matlabpool(myCluster, nslots)  % for MATLAB R2013a or older\r\ns = 0;\r\nparfor i=1:n\r\n  s = s + i;   % compute s = 1 + 2 + ... + n = n(n+1)\/2\r\nend\r\n\r\nfprintf(1,'Computed arithmetic sequence sum s = %d', s);\r\nfprintf(1,'  (correct answer: %d)\\n\\n\\n', n*(n+1)\/2);\r\n\r\n%matlabpool close  % MATLAB 2013a or older\r\ndelete(gcp)      % MATLAB 2014a or newer\r\n\r\nend<\/span><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>See <a href=\"#MULTIPLE\">Running Multiple Batch Jobs<\/a> on simple way to run a group of jobs.<\/em><\/p><\/blockquote>\n<p><\/div>\n<\/div>\n\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Running Pre-compiled MATLAB Standalone Jobs<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\">BU has a site license for MATALB, so the number of MATLAB jobs running at the same time is unlimited. It is recommended to run M-files rather than run standalone executables. But if you do need to run standalone executables for some reason, here is an example for how to do it via submitting a batch job,<\/p>\n<pre><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\">-t 100-300:200 .\/<a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/rcs-archive\/matlab-archive\/standalone\/#RUN_STANDALONE_JOB\">run_standalone_job<\/a><\/span><\/code><\/pre>\n<p><span style=\"display: inline !important; float: none; background-color: transparent; color: #333333; cursor: text; font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;\">See <\/span><a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/rcs-archive\/matlab-archive\/standalone\/\">How to Create &amp; Run MATLAB Standalone Executable<\/a><span style=\"display: inline !important; float: none; background-color: transparent; color: #333333; cursor: text; font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;\"> for details. <\/span><\/p>\n<p><\/div>\n<\/div>\n\n<div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Running Implicitly Parallel MATLAB Batch Jobs<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"><a name=\"THREADS\" id=\"THREADS\"><\/a>Unlike the Parallel Computing Toolbox (with which the programmer turns on parallelism manually by way of parallel utilities), implicitly parallel computations are automatically performed by MATLAB if it deems the computations worthwhile for parallelization. For example, many MATLAB vector operations \u2014 especially level-3, or \u039f(n<sup>3<\/sup>) \u2014 linear algebra operations such as matrix-matrix multiply or solvers for linear algebraic system of equations, could trigger MATLAB&#8217;s automatic (as opposed to multiprocessing with the PCT) multithreading feature if the amount of computation is significant. If you are certain that your MATLAB code can take advantage of <a href=\"https:\/\/www.bu.edu\/tech\/z-retired-pages\/matlab-pct\/implicit-parallelism\/\">implicit parallelism<\/a>, remove the <code>-singleCompThread<\/code> flag from <code><span class=\"placeholder\">mbatch<\/span><\/code> and submit the job to a multiprocessor queue. There are two situations associated with implicit parallel computation in regards to batch processing: using a whole node (recommended options are 28 or 16 cores) versus using a partial node (recommended options are 8 or 4 cores).<\/p>\n<ol>\n<li>Sample batch script for using a <em>whole node <\/em>(taking 28 cores for example),<em><\/em> <code><span class=\"placeholder\">wnbatch<\/span><\/code>:\n<pre class=\"code-block\"><code>#!\/bin\/bash -l\r\nmodule load matlab\r\nmatlab -nodisplay -r \"<em>A=rand(1e4); I=A*A; exit<\/em>\"<\/code><\/pre>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -pe omp 28<\/span> <span class=\"placeholder\">.\/wnbatch<\/span><\/code><\/pre>\n<p>In this example, a whole compute node with 28 cores is assigned to the job and the MATLAB functions are parallelly executed on the 28 cores.<\/li>\n<li>Sample batch script for using a <em>partial node<\/em> (taking 8 cores for example), <code><span class=\"placeholder\">pnbatch<\/span><\/code>:\n<pre class=\"code-block\"><code>#!\/bin\/bash -l \r\nmodule load matlab\r\nmatlab -nodisplay -r \"<em>maxNumCompThreads($NSLOTS); A=rand(1e4); I=A*A; exit<\/em>\"<\/code><\/pre>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -pe omp 8<\/span> <span class=\"placeholder\">.\/pnbatch<\/span><\/code><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>The <code><span class=\"command\">qsub<\/span><\/code> command line option <b>-pe omp 8<\/b> passes the core count to the batch scheduler&#8217;s $NSLOTS environment variable at runtime.<\/em><\/p><\/blockquote>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>Note the absence of the MATLAB option <b>-singleCompThread<\/b> in <code>pnbatch<\/code><\/em><\/p><\/blockquote>\n<p>In this example, 8 cores are assigned to the job and the MATLAB functions are parallelly executed on the 8 cores. Note that it is possible that the job is assigned to a compute node with more than 8 cores (such as 16 cores or 28 cores), but only 8 cores (i.e. a part of the compute node) are allowed to be used by the job. If the MATLAB program actually runs on more than 8 cores, the job will be killed automatically. As such, it is important to use the function <i><b>maxNumCompThreads($NSLOTS)<\/b> <\/i>to ensure that the thread number used in the MATLAB program is the same as the requested number of cores. <span style=\"display: inline !important; float: none; background-color: transparent; color: #333333; cursor: text; font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif; font-size: 16px; font-style: normal; font-variant: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration: none; text-indent: 0px; text-transform: none; -webkit-text-stroke-width: 0px; white-space: normal; word-spacing: 0px;\"> <\/span><\/p>\n<p><\/div>\n<\/div>\n\n<p><a name=\"MULTIPLE\" id=\"MULTIPLE\"><\/a><div class=\"bu_collapsible_container \" aria-live=\"polite\" data-customize-animation=\"false\"><h4 class=\"bu_collapsible\" aria-expanded=\"false\"tabindex=\"0\" role=\"button\">Running Multiple Batch Jobs With the qsub Array Job Option<\/h4><div class=\"bu_collapsible_section\" style=\"display: none;\"><a name=\"ARRAYJOB\" id=\"ARRAYJOB\"><\/a>Parametric studies (running an independent application multiple times by varying one or more parameters) are often necessary in many applications such as Monte Carlo simulations (in which a random number is a parameter) and image processing (for which the parameter is a reference to an image file). Towards this goal, the <code><span class=\"command\">qsub -t<\/span> <span class=\"placeholder\">#-#[:#]<\/span><\/code> (aka Array Job) option is ideally suited for parametric studies. Note that if the system load permits, jobs under a parametric study run concurrently \u2014 and effectively as Embarrassingly Parallel jobs \u2014 they practically scale linearly. The <code><span class=\"command\">qsub -t<\/span> <span class=\"placeholder\">#-#[:#]<\/span><\/span><\/code> option essentially has the effect of the PCT&#8217;s <code>parfor<\/code>. The <code><span class=\"command\">qsub<\/span><\/code> manpage states that &#8220;The option argument to -t specifies the number of array job tasks and the index number which will be associated with the tasks. The index numbers will be exported to the job tasks via the environment variable SGE_TASK_ID.&#8221; The relationship between SGE_TASK_ID and the -t input argument is demonstrated below, along with examples<\/p>\n<pre><code>% qsub -t SGE_TASK_FIRST <b>\u2014<\/b> SGE_TASK_LAST <b>:<\/b> SGE_TASK_STEP . . .\r\n% qsub generates SGE_TASK_ID for each job with above indices\r\n% Simulate SGE_TASK_ID with MATLAB colon syntax -- NOT REQUIRED\r\nSGE_TASK_ID=SGE_TASK_FIRST<b>:<\/b>SGE_TASK_STEP<b>:<\/b>SGE_TASK_LAST;\r\n           = [100,300,500]; % qsub -t 100-500:200 (run 3 jobs)\r\n           = 100;           % qsub -t 100 (run 1 job)\r\n           = [3,4,5];       % qsub -t 3-5 (default step size 1)<\/code><\/pre>\n<div class=\"highlight-yellow\"><b>Cautionary Notes:<\/b> While it is convenient to use the Array Job option, proceed with care when specifying the index range. An array job with <code>qsub -t 100-300<\/code> will spawn 201 batch jobs because the default step size is 1 ! Anticipatorily, name your array job<br \/>\n<code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\">-N myJobs -t 100-300:200 . . .<\/span><\/code><br \/>\nIn the event that an array job was submitted by mistake, simply delete all tasks with<br \/>\n<code><span class=\"prompt\">scc1$ <\/span><span class=\"command\">qdel -u yourUSERID<\/span> <span class=\"placeholder\"> \"myJobs*\"<\/span><\/code><\/div>\n<h6>Single-Core Parametric Studies<\/h6>\n<p>To perform a single-core parametric study, one could use <code>run_matlab_job<\/code> multiple times, each with a different value of <code><span class=command\">qsub<\/span> -v n=N ...<\/code> However, it is more convenient to run the jobs as an array, with which the environment variable $SGE_TASK_ID is used directly (or indirectly, like <code>n=fct($SGE_TASK_ID)<\/code>) as the random matrix order <code>n<\/code>.<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> <span class=\"placeholder\"> -t 3-7:2 .\/run_matlab_aj<\/span><\/code><\/pre>\n<p><code><span class=\"placeholder\">run_matlab_aj<\/span><\/code><\/p>\n<pre class=\"code-block\"><code>#!\/bin\/bash -l\r\n#$ -v alpha=1\r\n#$ -v beta=2\r\nmodule load matlab\r\n# use env var SGE_TASK_ID as random matrix size n\r\nmatlab -nodisplay -singleCompThread \\\r\n   -r \"myRand($SGE_TASK_ID, $alpha, $beta), exit\"<\/code><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>As previously explained in <a href=\"#RUN_MATLAB_JOB\"><code>run_matlab_job<\/code><\/a>, the double quoted segment in <code><span class=\"placeholder\">run_matlab_aj<\/span><\/code> is first processed by the system shell to replace all specified environment variables with their respective values before passing control on to MATLAB. The above example runs 3 jobs with $SGE_TASK_ID passed to each as 3, 5, and 7, respectively as the direct substitute for <code>n<\/code>, the random square matrix order.<\/em><\/p><\/blockquote>\n<pre class=\"code-block\"><code>function myRand(n, alpha, beta)\r\n% This is a companion to run_matlab_aj batch script to demonstrate\r\n% qsub -t option. It computes a random matrix, then save output of \r\n% task to a file with name indicative of task\r\nA = rand(n);                      % computes random matrix\r\nfilname=['output_' num2str(n)];    % name of file to be saved\r\nsave(filname, 'A','alpha','beta'); % saves A to a mat file<\/code><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>Above, <code>n<\/code> (= $SGE_TASK_ID) is also used to define I\/O file name for individual tasks to prevent all tasks writing to the same output file.<\/em><\/p><\/blockquote>\n<h6>Multi-Core (PCT) Parametric Studies<\/h6>\n<p>Here are example commands to submit batch jobs for multi-core parametric studies:<\/p>\n<pre class=\"code-block\"><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -t 128-176:16 -pe omp 16<\/span> <span class=\"placeholder\">.\/run_matlab_pct_aj # 16 cores; n=128, 144, 160, or 176<\/span>\r\n<span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -t 140-196:28<\/span> <span class=\"placeholder\">.\/run_matlab_pct_aj   # 28 cores; n=140, 168, or 196<\/span><\/pre>\n<p><b>run_matlab_pct_aj<\/b>:<\/p>\n<pre class=\"code-block\"><span class=\"placeholder\">#!\/bin\/bash -l\r\n#$ -pe omp 28\r\n<span style=\"color: #222222; font-family: Consolas;\">module load matlab<\/span>\r\n# Additional qsub options here . . .\r\nmatlab -nodisplay -r \"runBatchJob($SGE_TASK_ID, $NSLOTS); exit\"<\/span><\/pre>\n<blockquote style=\"border-left: 5px solid #eeeeee; margin: 15px 30px 0px 10px; padding-left: 20px;\"><p><em>Each job in the job array will run on a whole compute node with 16 or 28 cores, and all cores on the node are automatically used by the MATLAB functions. Using a whole node for one job has several advantages versus using part of a node: <\/em><\/p>\n<ol>\n<li>More cores may help to speedup computation.<\/li>\n<li>Being the sole user on the node guarantees that all of the node&#8217;s memory is available to your job.<\/li>\n<li>Having multiple PCT jobs running on the same node may cause a problem. This situation is avoided by using a whole node for one job.<\/li>\n<\/ol>\n<p>&nbsp;<\/p><\/blockquote>\n<h6>Parametric Studies Requiring Multiple Parameters<\/h6>\n<p>Here are a few ways to run parametric studies with multiple variables.<\/p>\n<ol>\n<li>Hold <code>N - 1<\/code> variables fixed (<i>e.g.,<\/i> <code>alpha, beta<\/code>) and run multiple jobs on the remaining <code>N<\/code>th variable via an Array Job\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -v alpha=1 -v beta=2 -t 10-15 <\/span> <span class=\"placeholder\">.\/run_matlab_aj<\/span>\r\n<span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub -v alpha=3 -v beta=4 -t 10-15 <\/span> <span class=\"placeholder\">.\/run_matlab_aj<\/span><\/code><\/pre>\n<\/li>\n<li>The above can be further automated, <i>e.g.,<\/i> with this <code><span class=\"placeholder\">runjobs<\/span><\/code> script:\n<pre class=\"code-block\"><code>#!\/bin\/bash -l\r\n# runjobs shell script\r\nfor a in `seq 1.1 0.3 3.2`; do\r\n  for b in `seq 1 1 3`; do\r\n    qsub -v alpha=$a -v beta=$b -t 10-15 run_matlab_aj\r\n  done\r\ndone<\/code><\/pre>\n<p>The above script uses the Unix sequence command <code><span class=\"command\">seq<\/span><\/code> to generate integer and floating-point sequences to feed the for-loop indices <code>a<\/code> and <code>b<\/code>. Input to <code><span class=\"command\">seq<\/span><\/code> has this format: <code> <span class=\"placeholder\">first:step:last<\/span><\/code>. Note that the symbol <b> ` <\/b> is a left-single-quote. In this case:<\/p>\n<pre class=\"code-block\"><code>a = [1.1, 1.4, 1.7, 2.0, 2.3, 2.6, 2.9, 3.2];   % seq 1.1 0.3 3.2\r\nb = [1, 2, 3];     % seq 1 1 3<\/code><\/pre>\n<p>Don&#8217;t forget to give <code><span class=\"placeholder\">runjobs<\/span><\/code> execute attribute:<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">chmod +x .\/runjobs<\/span><\/code><\/pre>\n<p>Then, executing <code><span class=\"placeholder\">runjobs<\/span><\/code> yields:<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">.\/runjobs<\/span>\r\nYour job-array 6826755.10-15:1 (\"runjobs\") has been submitted\r\n.  .  .  .  .\r\n.  .  .  .  .<\/code><\/pre>\n<\/li>\n<li>Alternatively, the MATLAB <code>ind2sub<\/code> utility may be used to map Array Job&#8217;s linear indexing to N-Dimensional indexing. For example, if you want to generate a 3&#215;4 array of 2-D indexing, you could submit an array job:\n<pre class=\"code-block\"><code><span class=\"prompt\">scc1$<\/span> <span class=\"command\">qsub<\/span> -t 1-12 . . .<\/code><\/pre>\n<p>This launches 12 jobs with their respective <code>$SGE_TASK_ID = 1, 2, 3, . . ., 12<\/code>. Next, in <code>myApp.m<\/code> :<\/p>\n<pre><code>% with SGE_TASK_ID (say, 7) passed into your myApp.m ...\r\n[i, j] = ind2sub([3 4], $SGE_TASK_ID)  % returns [row, col]\r\ni =\r\n     1\r\nj =\r\n     3\r\nalpha = Alpha(i);  % Alpha is independent of j\r\nbeta  = Beta(j);   % Beta is independent of I\r\n<\/code><\/pre>\n<\/li>\n<\/ol>\n<p><\/div>\n<\/div>\n<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This page demonstrates procedures for running MATLAB applications through the SCC&#8217;s batch scheduler. For MATLAB operations such as code development, GUI, and other graphical rendering, an interactive MATLAB window is the natural and preferred mode of operation. Other applications, such as running long duration production runs that do not require interaction, are best run in&#8230;<\/p>\n","protected":false},"author":1692,"featured_media":0,"parent":4606,"menu_order":1,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66367"}],"collection":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/users\/1692"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/comments?post=66367"}],"version-history":[{"count":51,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66367\/revisions"}],"predecessor-version":[{"id":140690,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66367\/revisions\/140690"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/4606"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/media?parent=66367"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}