It can be challenging to estimate how much memory your job will require before submission. Benchmarking tests are available for specific applications that can provide a guide but initially it is best to run your job and review the SCC’s job status reports. Each job is allocated virtual memory<\/b> throughout the job’s runtime. Virtual memory<\/b> is the required amount of memory for the job to run and can be accessed with three commands: The usage 1<\/b> line contains the maxvmem<\/b> which reports the maximum virtual memory that has been used during the cpu runtime. In this example, Note:<\/b> In this example, the compute node is VIRT<\/b> and RES<\/b> represents the total amount of allocated memory (virtual) and actual physical memory (resident) for each process, respectively. In this example, four MATLAB scripts are running in parallel: my_matrix1.m, my_matrix2.m, my_matrix3.m, and my_matrix4.m. Each of these processes has been allocated 10.7GB of memory. You would need to request a minimum of 44GB for four cores, or 11GB per core:<\/p>\n Note:<\/b> In this example, The slots<\/b> variable reports the number of cores requested for this job and the maxvmem<\/b> reports the maximum virtual memory used for this job. In this example, Estimating Memory Demands It can be challenging to estimate how much memory your job will require before submission. Benchmarking tests are available for specific applications that can provide a guide but initially it is best to run your job and review the SCC’s job status reports. Each job is allocated virtual memory throughout the job’s…<\/p>\n","protected":false},"author":1692,"featured_media":0,"parent":137962,"menu_order":7,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/137976"}],"collection":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/users\/1692"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/comments?post=137976"}],"version-history":[{"count":5,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/137976\/revisions"}],"predecessor-version":[{"id":147900,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/137976\/revisions\/147900"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/137962"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/media?parent=137976"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}qstat<\/span><\/code>, top<\/span><\/code>, and qacct<\/span><\/code>. qstat<\/span><\/code> and top<\/span><\/code> allow you to monitor your jobs’ processes in real time and qacct<\/span><\/code> is a full report available after a job has finished. Guidelines for submitting batch jobs with large memory requirements are available here<\/a>.<\/p>\n\n
<\/a>qstat<\/h2>\n
qstat<\/span><\/code> is an SGE command that reports the status of jobs submitted to the cluster. To see more details of a specific job running on the cluster, you will need to run qstat<\/span><\/code> with the -j<\/span> job_ID<\/span><\/code> flag specifying the job_ID<\/span><\/code> assigned to your job which can be be found by qstat -u<\/span> userID<\/span><\/code>.<\/p>\nscc % <\/span>qstat -u <\/span>userID<\/span>\r\njob-ID prior name user state submit\/start at queue slots ja-task-ID\r\n-----------------------------------------------------------------------------------------------------------------\r\n4717015<\/span><\/span> 0.10072 my_job1 userID r 03\/01\/2018 09:35:08 p8@scc-pf2.scc.bu.edu 8\r\n4717016 0.10072 my_job2 userID r 03\/01\/2018 09:35:08 p8@scc-pf2.scc.bu.edu 8\r\n<\/span>\r\nscc % <\/span>qstat <\/span>-j 4717015<\/span>\r\n==============================================================\r\njob_number: 4717015\r\nexec_file: job_scripts\/4717016\r\nsubmission_time: Thu Mar 1 09:34:35 2018\r\nowner: userID\r\n...\r\njob_name: my_job1\r\nstdout_path_list: NONE:NONE:\/projectnb\/scv\/userID\/scripts\/log\/\r\njobshare: 0\r\nenv_list: PATH=\/projectnb\/scv\/userID\/scripts:\/bin:\/usr\/bin:\/usr\/local\/sbin:\/usr\/sbi\r\njob_args: sub001\r\nscript_file: my_job1.qsub\r\nparallel environment: omp8 range: 8\r\nverify_suitable_queues: 2\r\nproject: scv\r\nusage 1:<\/span> cpu=7:58:19, mem=5319.88221 GBs, io=66.36036, vmem=15.690G, maxvmem=15.886G<\/span><\/span>\r\nscheduling info: (Collecting of scheduler job information is turned off)\r\n<\/span><\/code><\/pre>\nmy_job1<\/span><\/code> requires 16GB of total memory during the first 8 hours of runtime.<\/p>\n<\/a>top<\/h2>\n
top<\/span><\/code> is a command that shows the active processes on a system. In order to see your active processes on the compute node your job is running on, you will need to run top<\/span><\/code> on that compute node. We can do this by remotely accessing the compute node running your job using ssh<\/span><\/code>. The compute node running your job can be identified using the qstat -u<\/span> userID<\/span><\/code> command.<\/p>\nscc % <\/span>qstat -u<\/span> userID<\/span>\r\njob-ID prior name user state submit\/start at queue slots ja-task-ID\r\n-----------------------------------------------------------------------------------------------------------------\r\n4717015 0.10072 my_job1 userID r 03\/01\/2018 09:35:08 p8@scc-pf2<\/span><\/span>.scc.bu.edu 8\r\n4717016 0.10072 my_job2 userID r 03\/01\/2018 09:35:08 p8@scc-pf2.scc.bu.edu 8\r\n<\/span>\r\nscc % <\/span>ssh -t <\/span>scc-pf2 <\/span>'top -u <\/span>userID<\/span>'\r\ntop - 14:37:07 up 40 days, 16:19, 7 users, load average: 0.11, 0.21, 0.14\r\nTasks: 418 total, 2 running, 416 sleeping, 0 stopped, 0 zombie\r\nCpu(s): 1.3%us, 0.1%sy, 0.0%ni, 98.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st\r\nMem: 132064072k total, 110925644k used, 21138428k free, 358992k buffers\r\nSwap: 8388604k total, 33376k used, 8355228k free, 107315220k cached\r\n\r\n PID USER PR NI VIRT RES <\/span> SHR S %CPU %MEM TIME+ COMMAND\r\n 37182 userID 20 0 13396 1416 <\/span> 852 R 3.9 0.0 0:00.03 top\r\n 36370 userID 20 0 77648 4756 <\/span>1080 S 0.0 0.0 0:01.99 sshd\r\n 36510 userID 20 0 10.7g 2.1g <\/span> 78m S 0.0 1.3 0:22.97 my_matrix1.m\r\n 36510 userID 20 0 10.7g 2.1g <\/span> 78m S 0.0 1.3 0:22.97 my_matrix2.m\r\n 36510 userID 20 0 10.7g 2.1g <\/span> 78m S 0.0 1.3 0:22.97 my_matrix3.m\r\n 36510 userID 20 0 10.7g 2.1g <\/span> 78m S 0.0 1.3 0:22.97 my_matrix4.m\r\n 36371 userID 20 0 9676 1916 <\/span>1384 S 0.0 0.0 0:00.03 bash\r\n 36475 userID 20 0 30944 5456 <\/span>2708 S 0.0 0.0 0:00.08 fslwish8.4\r\n 36502 userID 20 0 13432 1232 <\/span> 904 S 0.0 0.0 0:00.05 freeview\r\n 36504 userID 20 0 600m 59m <\/span> 34m S 0.0 0.0 0:05.91 freeview.bin\r\n 36510 userID 20 0 3872m 412m <\/span> 78m S 0.0 0.3 0:22.97 MATLAB\r\n 37181 userID 20 0 92872 1840 <\/span> 872 S 0.0 0.0 0:00.00 sshd\r\n<\/span><\/code><\/pre>\nscc-pf2<\/span><\/code><\/span> which will need to be changed to the compute node allocated to your job. This is reported in the ‘queue’ column of qstat -u<\/span> userID<\/span><\/code> command.<\/em><\/p><\/blockquote>\n#!\/bin\/bash -l<\/span>\r\n#$ -P my_project<\/span>\r\n#$ -N my_matlab_job<\/span>\r\n#$ -l mem_per_core=11G<\/span><\/span>\r\n#$ -pe omp 4<\/span><\/span><\/code><\/pre>\n<\/a>qacct<\/h2>\n
scc % <\/span>qacct <\/span>-o userID <\/span>-d 1 <\/span> -j\r\n==============================================================\r\nqname p-int\r\nhostname scc-pi2.scc.bu.edu\r\ngroup scv\r\nowner userID\r\nproject scv\r\ndepartment defaultdepartment\r\njobname my_job\r\njobnumber 4035924\r\n...\r\nqsub_time Thu Jan 25 14:45:36 2018\r\nstart_time Thu Jan 25 14:46:15 2018\r\nend_time Fri Jan 26 02:46:16 2018\r\ngranted_pe NONE\r\nslots 8<\/span><\/span>\r\n...\r\ncpu 202.390\r\nmem 7478.277\r\nio 0.348\r\niow 0.000\r\nmaxvmem 63.953G<\/span><\/span>\r\n...\r\n<\/span><\/code><\/pre>\n-d<\/span><\/code> is the number of days of job summaries you want to view. See man qacct<\/span><\/code> for more details.<\/span><\/em><\/p><\/blockquote>\nmy_job<\/span><\/code> would need to request 64GB of total memory, or 8GB per core to run optimally:<\/p>\n#!\/bin\/bash -l<\/span>\r\n#$ -P my_project<\/span>\r\n#$ -N my_job<\/span>\r\n#$ -l mem_per_core=8G<\/span><\/span>\r\n#$ -pe omp 8<\/span><\/span><\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"