{"id":66195,"date":"2013-05-14T14:57:18","date_gmt":"2013-05-14T18:57:18","guid":{"rendered":"http:\/\/www.bu.edu\/tech\/?page_id=66195"},"modified":"2022-05-27T13:57:47","modified_gmt":"2022-05-27T17:57:47","slug":"modules","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/tech\/support\/research\/software-and-programming\/software-and-applications\/modules\/","title":{"rendered":"Module Basics"},"content":{"rendered":"<p>The <code><span class=\"command\">module<\/span><\/code> package is available on the <a href=\"https:\/\/www.bu.edu\/tech\/support\/research\/computing-resources\/scc\/\">Shared Computing Cluster<\/a>, allowing users to access non-standard tools or alternate versions of standard packages. This is also an alternative way to configure your environment as required by certain packages. Specific modules can be loaded and unloaded as required. The module command is provided by the <a href=\"https:\/\/www.tacc.utexas.edu\/research-development\/tacc-projects\/lmod\">Lmod<\/a> software, developed at the Texas Advanced Computing Center.<\/p>\n<p>The version of Python that is available without loading a module is an older version (2.7.5) that is intended for system usage only. Load a module in order to use Python.<\/p>\n<h3>Table of Contents<\/h3>\n<ul>\n<li><a href=\"#Commands\">Common Commands<\/a><\/li>\n<li><a href=\"#Using\">Using Common Commands<\/a><\/li>\n<li><a href=\"#Basic\">Basic Command Sequence<\/a><\/li>\n<li><a href=\"#Scripts\">Scripts and Batch Commands<\/a><\/li>\n<li><a href=\"#rc\">Modules Best Practices<\/a><\/li>\n<li><a href=\"#redirect\">How to grep &#8216;module avail&#8217;\/How to redirect &#8216;module&#8217; output?<\/a><\/li>\n<\/ul>\n<hr \/>\n<p><a name=\"Commands\"><\/a><\/p>\n<h2>Common Commands<\/h2>\n<table border=\"0\">\n<tbody>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px;\">module list<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">List currently loaded modules.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px;\">module avail<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">List available packages.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px;\">module spider<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">List available packages in a different format.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px; white-space: nowrap;\">module help [<em>modulefile<\/em>]<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">Description of specified module.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px; white-space: nowrap;\">module show [<em>modulefile<\/em>]<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">Displays information about specified module, including environment changes, dependencies, software version and path.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px; white-space: nowrap;\">module load [<em>modulefile<\/em>]<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">Loads module or specifies which dependencies have not been loaded.<\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px; white-space: nowrap;\">module unload [<em>modulefile<\/em>]<\/span><\/td>\n<td><span style=\"font-size: 13px; line-height: 19px;\">Unloads specified module from environment.<\/span><span style=\"font-size: 13px; line-height: 19px;\"> <\/span><\/td>\n<\/tr>\n<tr>\n<td><span style=\"font-size: 13px; line-height: 19px;\">module purge <\/span><\/td>\n<td>Unloads all loaded modules<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>More information can be found in the <a href=\"http:\/\/modules.sourceforge.net\/man\/module.html\">Module man page<\/a>.<\/p>\n<hr \/>\n<p><a name=\"Using\"><\/a><\/p>\n<h2>Using Common Commands<\/h2>\n<h3>module list<\/h3>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module list<\/span>\r\n<span class=\"output\">No Modulefiles Currently Loaded.<\/span>\r\n<\/code><\/pre>\n<h3>module avail<\/h3>\n<p>List <strong>all<\/strong> available packages.<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module avail<\/span>\r\n\r\n<span class=\"output\">--------------------------------- \/share\/module.7\/bioinformatics --------------------------------- \r\n   2brad_denovo\/2019-01-22_giteec5016            meshclust2\/2.1.0\r\n   2brad_gatk\/2019-01-22_git1fcc9e8              metal\/2010-02-08\r\n   admixture\/1.3.0                               metal\/2011-03-25                (D)\r\n   angsd\/0.923                                   metalge\/2010-02-08\r\n   annovar\/2018apr                               minimac2\/2014-09-15\r\n   artemis\/18.0.3                                minimac3\/2.0.1\r\n   augustus\/3.3.2                                minimac4\/1.0.0\r\n   bamtools\/2.5.1                                mirdeep2\/0.1.0\r\n   bamutil\/1.0.14                                mixcr\/3.0.3\r\n   basemount\/0.15.103.3011                       mmap\/2018-04-07\r\n   basespace-cli\/0.8.12.590                      morgan\/3.2\r\n   basespace-cli\/0.9.17                          morgan\/3.4                      (D)\r\n   basespace-cli\/0.10.8                   (D)    mosdepth\/0.2.6\r\n   bayescan\/2.1                                  mothur\/1.35.0\r\n   bbmap\/38.16                                   multiqc\/1.6\r\n   bcbio\/1.1.1                                   mummer\/3.23\r\n<\/span>\r\n<\/code><\/pre>\n<p>You can also search for packages by name. For example to see all versions of &#8216;tensorflow&#8217;, you can search for the word &#8216;tensorflow&#8217; or a substring like &#8216;tensor&#8217;,<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module avail<\/span> <span class=\"placeholder\">tensor\r\n\r\n<span class=\"output\">--------------------------------- \/share\/module.7\/machine-learning ------------------------------------------ \r\n   tensorflow\/1.12    tensorflow\/1.13.1    tensorflow\/1.15.0 (D)    tensorflow\/2.0.0\r\n\r\n  Where:\r\n   D:  Default Module\r\n<\/span><\/span><\/code><\/pre>\n<h3>module spider<\/h3>\n<p>List available packages in a different format.<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module spider<\/span>\r\n\r\n<span class=\"output\">-----------------------------------------------------------------------------------------------------------\r\nThe following is a list of the modules currently available:\r\n-----------------------------------------------------------------------------------------------------------\r\n  2brad_denovo: 2brad_denovo\/2019-01-22_giteec5016\r\n\r\n  2brad_gatk: 2brad_gatk\/2019-01-22_git1fcc9e8\r\n\r\n  3d-ice: 3d-ice\/2.2.6\r\n\r\n  R: R\/3.0.0, R\/3.5.1, R\/3.6.0_intel-2018, R\/3.6.0, R\/3.6.1_intel-2019\r\n\r\n  admixture: admixture\/1.3.0\r\n\r\n  afni: afni\/17.0.18-omp, afni\/19.1.00-omp, afni\/19.1.00, afni\/2011-12-21.1014-xorg, afni\/2011-12-21.1014\r\n\r\n  amber: amber\/16\r\n\r\n  amgx: amgx\/2019-12-13_gitb3101ff\r\n    AmgX is a GPU accelerated core solver library that speeds up computationally intense linear solver\r\n    portion of simulations.\r\n\r\n  anaconda2: anaconda2\/5.2.0\r\n\r\n  anaconda3: anaconda3\/5.2.0\r\n<\/span>\r\n<\/code><\/pre>\n<p>You can also search for packages by name. For example to see all versions of &#8216;tensorflow&#8217;, you can search for the word &#8216;tensorflow&#8217; or a substring like &#8216;tensor&#8217;,<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module spider<\/span> <span class=\"placeholder\">tensor\r\n\r\n<span class=\"output\">-----------------------------------------------------------------------------------------------------------\r\n  tensorflow:\r\n-----------------------------------------------------------------------------------------------------------\r\n     Versions:\r\n        tensorflow\/1.12\r\n        tensorflow\/1.13.1\r\n        tensorflow\/1.15.0\r\n        tensorflow\/2.0.0\r\n\r\n-----------------------------------------------------------------------------------------------------------\r\n  For detailed information about a specific \"tensorflow\" module (including how to load the modules) use the module's full name.\r\n  For example:\r\n\r\n     $ module spider tensorflow\/2.0.0\r\n-----------------------------------------------------------------------------------------------------------\r\n\r\n<\/span><\/span><\/code><\/pre>\n<h3>module help [<em>modulefile<\/em>]<\/h3>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module help<\/span> <span class=\"placeholder\">gcc<\/span>\r\n<span class=\"output\">------------------------------------ Module Specific Help for \"gcc\/8.3.0\" -------------------------------------\r\n\r\ngcc 8.3.0: GNU Compiler Collection (GCC)\r\n\r\nC, C++, and Fortran compilers.\r\n\r\nFor more information on gcc, please see https:\/\/gcc.gnu.org\/\r\n\r\nThe following environment variables are provided for convenience:\r\n$SCC_GCC_DIR -- Base Package Directory\r\n$SCC_GCC_BIN -- Package Executables Directory\r\n$SCC_GCC_LIB -- Package Libraries Directory\r\n$SCC_GCC_LICENSE -- Package License Information\r\n<\/span><\/code><\/pre>\n<h3>module show [<em>modulefile<\/em>]<\/h3>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module show<\/span> <span class=\"placeholder\">gcc<\/span>\r\n<span class=\"output\">-----------------------------------------------------------------------------------------------------------\r\n   \/share\/module.7\/programming\/gcc\/8.3.0.lua:\r\n-----------------------------------------------------------------------------------------------------------\r\nhelp([[\r\ngcc 8.3.0: GNU Compiler Collection (GCC)\r\n\r\nC, C++, and Fortran compilers.\r\n\r\nFor more information on gcc, please see https:\/\/gcc.gnu.org\/\r\n\r\nThe following environment variables are provided for convenience:\r\n$SCC_GCC_DIR -- Base Package Directory\r\n$SCC_GCC_BIN -- Package Executables Directory\r\n$SCC_GCC_LIB -- Package Libraries Directory\r\n$SCC_GCC_LICENSE -- Package License Information\r\n\r\n]])\r\nwhatis(\"Name:         gcc\")\r\nwhatis(\"Version:      8.3.0\")\r\nwhatis(\"Description:  GNU Compiler Collection (GCC)\")\r\nwhatis(\"URL:          https:\/\/gcc.gnu.org\/\")\r\nwhatis(\"Categories:   programming\")\r\nwhatis(\"Keywords:     centos7, programming, gnu, compiler, c, c++, fortran\")\r\nsetenv(\"SCC_GCC_DIR\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\")\r\nsetenv(\"SCC_GCC_BIN\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/bin\")\r\nsetenv(\"SCC_GCC_LIB\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/lib64\")\r\nsetenv(\"SCC_GCC_LIB32\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/lib\")\r\nsetenv(\"SCC_GCC_LICENSE\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/COPYING\")\r\nprepend_path(\"PATH\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/bin\")\r\nprepend_path(\"LD_LIBRARY_PATH\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/lib\")\r\nprepend_path(\"LD_LIBRARY_PATH\",\"\/share\/pkg.7\/gcc\/8.3.0\/install\/lib64\")\r\n<\/span>\r\n<\/code><\/pre>\n<hr \/>\n<p><a name=\"Basic\"><\/a><\/p>\n<h2>Basic Command Sequence<\/h2>\n<p>Below is a basic sequence of module commands to use to get started using modules on the SCC.<\/p>\n<ol>\n<li>List all available modules by executing &#8220;<code><span class=\"command\">module avail<\/span><\/code>&#8221; command.  The results are a list of module names that can be loaded.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module avail<\/span>\r\n<span class=\"output\">---------------------------- \/share\/etc\/modulefiles ----------------------------\r\n2brad_denovo\/2019-01-22_giteec5016            forestpmplot\/1.0.1                        plink\/1.07                        (D)\r\n2brad_gatk\/2019-01-22_git1fcc9e8              gatk\/3.8-1                                plink\/1.90b6.4\r\nadmixture\/1.3.0                               gatk\/4.0.11.0                             plink\/2.00a1LM\r\nangsd\/0.923                                   gatk\/4.0.12.0                             plink\/2.00a2.3\r\nannovar\/2018apr                               gatk\/4.1.2.0                       (D)    plink\/2.0\r\nannovar\/2019oct24                      (D)    gatk\/4.1.3.0                              plinkseq\/0.10\r\nanvio\/6.1                                     gatk\/4.1.4.1                              postgap\/2020-05-11-git0453c91\r\n...                                   ...\r\n<\/span><\/code><\/pre>\n<\/li>\n<li>Filter the result by adding keywords to &#8220;<code><span class=\"command\">module avail<\/span><\/code>&#8221; command, such as &#8220;<code><span class=\"command\">julia<\/span><\/code>&#8220;.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module avail julia<\/span>\r\n<span class=\"output\">------------------------- \/share\/module.7\/programming --------------------------\r\n   julia\/0.7.0    julia\/1.0.2    julia\/1.3.0    julia\/1.4.2    julia\/1.5.0 (D)\r\n\r\n  Where:\r\n   D:  Default Module\r\n\r\nUse \"module spider\" to find all possible modules.\r\nUse \"module keyword key1 key2 ...\" to search for all possible modules matching\r\nany of the \"keys\".\r\n<\/span><\/code><\/pre>\n<\/li>\n<li>Load the module using &#8220;<code><span class=\"command\">module load<\/span><\/code>&#8221; and append the module name.  The following example shows loading Julia version 1.5.0.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module load julia\/1.5.0<\/span>\r\n<\/code><\/pre>\n<\/li>\n<li>Confirm the module is loaded by executing &#8220;<code><span class=\"command\">module list<\/span><\/code>&#8220;.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module list<\/span><br>\r\n<span class=\"output\">Currently Loaded Modules:\r\n  1) julia\/1.5.0\r\n\r\n<\/span><\/code><\/pre>\n<\/li>\n<li>Now you can run your application. In this example we will check the version of Julia loaded.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">julia -version<\/span>\r\n<span class=\"output\">julia version 1.5.0\r\n<\/span><\/code><\/pre>\n<\/li>\n<li>When you are done with the application, unload the module using &#8220;<code><span class=\"command\">module unload<\/span><\/code>&#8221; command with the name of the module you want to remove.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module unload julia\/1.5.0<\/span>\r\n<\/code><\/pre>\n<\/li>\n<li> Confirm the module was unloaded by running &#8220;<code><span class=\"command\">module list<\/span><\/code>&#8221; command.\n<pre class=\"code-block\"><code>[<span class=\"placeholder\">username<\/span>@scc1 ~]$ <span class=\"command\">module list<\/span>\r\n<span class=\"output\">No modules loaded\r\n<\/span><\/code><\/pre>\n<\/li>\n<\/ol>\n<hr \/>\n<p><a name=\"Scripts\"><\/a><\/p>\n<h2>Scripts and Batch Commands<\/h2>\n<p>To ensure that modules are loaded correctly within a script file add <code><span class=\"command\">-l<\/span><\/code> option to the first line of your script, i.e.:<\/p>\n<pre class=\"code-block\"><code><span class=\"command\">#!\/bin\/bash -l <\/span><\/code><\/pre>\n<p>When including modules in scripts and batch submission commands (via <code><span class=\"command\">qsub<\/span><\/code>), it is the best practice to specify modules and applications by version number. Over time, newer versions of the applications will be installed. Specifying versions of applications ensures that in the future jobs will run the same way as they do now.<\/p>\n<p>Example:<\/p>\n<pre class=\"code-block\"><code><span class=\"command\">module load<\/span> <span class=\"placeholder\">tophat\/tophat-2.0.4_gnu446<\/span><\/code><\/pre>\n<p>rather than<\/p>\n<pre class=\"code-block\"><code><span class=\"command\">module load<\/span> <span class=\"placeholder\">tophat<\/span><\/code><\/pre>\n<hr \/>\n<p><a name=\"rc\"><\/a><\/p>\n<h2>Modules Best Practices<\/h2>\n<p>Below are best practices for using modules<\/p>\n<ul>\n<li>RCS is always adding new and updated software modules to the SCC, but newer versions are not always backwards compatible and can break existing scripts or workflows.  When using the &#8220;<code><span class=\"command\">module load<\/span><\/code>&#8221; command, one should always use the full module name (including version number) to ensure the same modules are loaded each time you work on your analysis.<\/li>\n<li>In order to reduce the chance of unexpected behavior caused by module conflicts, only load modules that are needed for the current script or workflow you are running.<\/li>\n<li>Avoid including &#8220;<code><span class=\"command\">module load<\/span><\/code>&#8221; commands in your <code>.bashrc<\/code> or <code>.bash_profile<\/code> files for convenience, as we have found this causes instability and other issues for our researchers.  As an alternative, create a bash script with the module load commands and source it each time, to load the modules needed.<\/li>\n<\/ul>\n<hr \/>\n<p><a name=\"redirect\"><\/a><\/p>\n<h2>How to redirect module output<\/h2>\n<p>This answer also addresses: <strong>How do I grep &#8216;module avail&#8217;?<\/strong> The short answer is: redirect standard error (<strong>stderr<\/strong>) to standard output (<strong>stdout<\/strong>). If you are using the bash shell, this is done using the &#8216;<code><span class=\"command\">2&gt;&amp;1<\/span><\/code>&#8216; syntax. For example, to search for <code><span class=\"placeholder\">python<\/span><\/code> in the available modules, you can type:<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">module -t avail 2&gt;&amp;1 | grep -i<\/span> <span class=\"placeholder\">python<\/span>\r\n<\/code><\/pre>\n<p>In [t]csh, the command is a bit more complicated, because there is no trivial way to redirect <strong>stderr<\/strong>. The following example calls the module command in a separate process and then it pipes the <strong>stderr<\/strong> and <strong>stdout<\/strong> to <code><span class=\"command\">grep<\/span><\/code>:<\/p>\n<pre class=\"code-block\"><code><span class=\"prompt\">scc4%<\/span> <span class=\"command\">(module -t avail) |&amp; grep -i<\/span> <span class=\"placeholder\">python<\/span>\r\n<\/code><\/pre>\n<div class=\"highlight-yellow\"><strong>Note<\/strong>: <code><span class=\"command\">module avail<\/span><\/code> already has built-in search capabilities which avoids most of the need to use tools like grep.<\/div>\n","protected":false},"excerpt":{"rendered":"<p>The module package is available on the Shared Computing Cluster, allowing users to access non-standard tools or alternate versions of standard packages. This is also an alternative way to configure your environment as required by certain packages. Specific modules can be loaded and unloaded as required. The module command is provided by the Lmod software,&#8230;<\/p>\n","protected":false},"author":1692,"featured_media":0,"parent":76894,"menu_order":1,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66195"}],"collection":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/users\/1692"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/comments?post=66195"}],"version-history":[{"count":51,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66195\/revisions"}],"predecessor-version":[{"id":153457,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/66195\/revisions\/153457"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/76894"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/media?parent=66195"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}