{"id":1863,"date":"2020-07-06T13:09:27","date_gmt":"2020-07-06T17:09:27","guid":{"rendered":"https:\/\/www.bu.edu\/engit\/?page_id=1863"},"modified":"2025-12-10T08:39:13","modified_gmt":"2025-12-10T13:39:13","slug":"lumerical","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/engit\/knowledge-base\/grid\/software\/lumerical\/","title":{"rendered":"Lumerical"},"content":{"rendered":"<h1>Prerequisites<\/h1>\n<ul>\n<li><b>Your username must be in an appropriate list on the Lumerical license server for you to receive a license.<\/b> See <a href=\"https:\/\/www.bu.edu\/engit\/services\/software\/\">here<\/a> for more information about setting up licensing for Lumerical.<\/li>\n<li>You must already be using the\u00a0<a href=\"http:\/\/www.bu.edu\/tech\/support\/research\/computing-resources\/scc\/\">SCC<\/a> and successfully running test jobs.<\/li>\n<li>You must be set up to run <a href=\"http:\/\/www.bu.edu\/tech\/support\/research\/software-and-programming\/programming\/multiprocessor\/\">MPI jobs with OpenMPI<\/a>. (That&#8217;s covered here, but the link has the basic setup, before Lumerical.)<\/li>\n<\/ul>\n<p><b>Parallel Lumerical Example<\/b><br \/>\nFirst, ssh to the SCC and then run:<\/p>\n<pre class=\"darkSnippet\">export PATH=\/ad\/eng\/bin:\/ad\/eng\/bin\/64\/:$PATH\r\nlumerical-fdtd-config-license<\/pre>\n<p>Enter these options in the window:<\/p>\n<ul>\n<li>Server: lumericallm.bu.edu<\/li>\n<li>Port: Default<\/li>\n<li>Configure redundant servers: Unchecked<\/li>\n<\/ul>\n<p>OR you can use modules, like so:<\/p>\n<pre class=\"darkSnippet\">module use -s \/ad\/eng\/etc\/modulefiles\r\nmodule load lumerical\/license<\/pre>\n<p>If you are using lumerical device instead of lumerical fdtd, you will also have to load the lumerical\/device module.<\/p>\n<h1>Basic Test<\/h1>\n<p>Now, we can run any MPI program we choose on any queue that has the &#8220;mpi&#8221; parallel environment configured, and Grid Engine will dynamically allocate hosts to the PE. There are many ways to invoke this and many options that you can pass, but here&#8217;s a simple example using Lumerical ompi test.<\/p>\n<p>First, the job script fdtd-mpi-example.sh is this:<\/p>\n<pre class=\"darkSnippet\"># This \"cwd\" line will make sure to run from inside the current directory\r\n#$ -cwd\r\n# This select Parallel Environment \"mpi\" with 4 nodes\r\n#$ -pe mpi 4\r\nhostname\r\ndate\r\nmpirun --mca btl_tcp_if_include eth0 -np $NSLOTS \/ad\/eng\/opt\/64\/lumerical\/mpitest\/cpi-ompi-lcl<\/pre>\n<p>Then, to run the job:<\/p>\n<pre class=\"darkSnippet\">scc ~ $ qsub job.sh\r\nYour job 1886511 (\"fdtd-mpi-example.sh\") has been submitted<\/pre>\n<p>Use &#8220;qstat&#8221; (or qmon) to see the job waiting, then running, and once it&#8217;s finished, you should have several files in your output directory, including a .o &#8220;output&#8221; file that looks something like this:<\/p>\n<pre class=\"darkSnippet\">scc ~ $ more job.sh.o1886511\r\nWarning: no access to tty (Bad file descriptor).\r\nThus no job control in this shell.\r\nnode\r\nWed Apr 27 05:32:25 EDT 2018\r\npi is approximately 3.1416009869231249, Error is 0.0000083333333318\r\nwall clock time = 0.040293<\/pre>\n<p>Apparently, Lumerical puts its information about the slave nodes into the .e file (standard error) instead of .o (standard output) for some reason.<\/p>\n<p>Notice that the part of our code that printed the hostname and then the date is running on the node that Grid Engine designated as the MPI master, and then the other four hostnames were printed in the .e file by mpirun&#8217;s invocation of cpi-ompi-lcl on the 4 slaves (one of which is the same as the master, in this case).<\/p>\n<h1>Full Test<\/h1>\n<p>Now, we can try a full test using paralleltest.fsp from Lumerical&#8217;s examples directory. Copy the file into your current directory:<\/p>\n<pre class=\"darkSnippet\">scc ~ $ cp \/ad\/eng\/opt\/64\/lumerical\/examples\/paralleltest.fsp<\/pre>\n<p>and change the last line of the script to:<\/p>\n<pre class=\"darkSnippet\">mpirun --mca btl_tcp_if_include eth0 -np $NSLOTS \/ad\/eng\/opt\/64\/lumerical\/bin\/fdtd-engine-ompi-lcl -logall paralleltest.fsp<\/pre>\n<p>And now we can submit it:<\/p>\n<pre class=\"darkSnippet\">scc ~ $ qsub fdtd-mpi-paralleltest.sh\r\nYour job 1886512 (\"fdtd-mpi-paralleltest.sh\") has been submitted<\/pre>\n<p>Using &#8220;qstat&#8221; we can see it running for about 2 minutes, and then we can check the .o and .e files:<\/p>\n<pre class=\"darkSnippet\">scc ~$ more fdtd-mpi-paralleltest.sh.o1886512\r\nWarning: no access to tty (Bad file descriptor).\r\nThus no job control in this shell.\r\nnode\r\nWed Apr 27 05:57:10 EDT 2011\r\n7% complete. Max time remaining: 1 min, 56 secs. Auto Shutoff: 1\r\n9% complete. Max time remaining: 1 min, 54 secs. Auto Shutoff: 1\r\n[SNIP]\r\n100% complete. Max time remaining: 0 secs. Auto Shutoff: 1<\/pre>\n<p>Trying this again with 16 processors instead of 4, we got:<\/p>\n<pre class=\"darkSnippet\">scc ~$ more fdtd-mpi-paralleltest.sh.o1886514\r\nWarning: no access to tty (Bad file descriptor).\r\nThus no job control in this shell.\r\nnode\r\nWed Apr 27 06:07:55 EDT 2018\r\n10% complete. Max time remaining: 34 secs. Auto Shutoff: 1\r\n[SNIP]\r\n100% complete. Max time remaining: 0 secs. Auto Shutoff: 1<\/pre>\n<p>So, four times as fast, using 16P instead of 4P. Looks like a linear speedup! You might as well experiment a lot to see where your speedup starts to fall off, if at all.<\/p>\n<h1>Scripting with .lsf files<\/h1>\n<p>You can find the example scripts at \/ad\/eng\/opt\/64\/lumerical\/eng-grid\/ :<\/p>\n<pre class=\"darkSnippet\">fdtd-mpi-eth.sh\r\nfdtd-pbs-template.sh\r\nfdtd-pfdtd-run-pbs.sh\r\nfdtd-run-pbs.sh<\/pre>\n<p>For sge, blocking uses the &#8220;-sync y&#8221; option to qsub, to block the qsub command line from returning until the job is actually done. And don&#8217;t follow that page&#8217;s suggestions about using the &#8220;fdtd-solutions&#8221; binary to run the lsf files &#8212; just do it through the GUI in CAD. (The fdtd-solutions binary seems to segfault, but the way you were always doing it in CAD works fine.)<br \/>\nWe set this all up in your CAD configuration as shown in the first screenshot on the web page mentioned above, but with the &#8220;Set FDTD engine&#8221; field set to:<\/p>\n<pre class=\"darkSnippet\">\/ad\/eng\/opt\/64\/lumerical\/eng-grid\/fdtd-run-pbs.sh<\/pre>\n<p>And then click Save, and then change the old &#8220;Local Host&#8221; resource to &#8220;Active&#8211;&gt;No&#8221; so that &#8220;run parallel FDTD&#8221; will by default qsub the job to the SCC.<\/p>\n<h1>See Also<\/h1>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li><a href=\"https:\/\/support.lumerical.com\/hc\/en-us\/articles\/360024974033-Linux-command-line\">Running from the Linux command line<\/a><\/li>\n<li><a href=\"https:\/\/kx.lumerical.com\/t\/running-simulations-in-the-background-on-linux\/3380\">Running simulations in the background onr linux<\/a><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Prerequisites Your username must be in an appropriate list on the Lumerical license server for you to receive a license. See here for more information about setting up licensing for Lumerical. You must already be using the\u00a0SCC and successfully running test jobs. You must be set up to run MPI jobs with OpenMPI. (That&#8217;s covered [&hellip;]<\/p>\n","protected":false},"author":1470,"featured_media":0,"parent":138,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/1863"}],"collection":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/users\/1470"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/comments?post=1863"}],"version-history":[{"count":6,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/1863\/revisions"}],"predecessor-version":[{"id":2966,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/1863\/revisions\/2966"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/pages\/138"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/engit\/wp-json\/wp\/v2\/media?parent=1863"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}