{"id":150138,"date":"2024-01-29T13:10:17","date_gmt":"2024-01-29T18:10:17","guid":{"rendered":"http:\/\/www.bu.edu\/tech\/?page_id=150138"},"modified":"2025-03-03T16:27:06","modified_gmt":"2025-03-03T21:27:06","slug":"building","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/tech\/support\/research\/software-and-programming\/containers\/building\/","title":{"rendered":"Building Containers"},"content":{"rendered":"<p><strong>Sections<\/strong><\/p>\n<ul>\n<li><a href=\"#import\">Building From Docker or Singularity Hub<\/a><\/li>\n<li><a href=\"#build\">Building From Scratch<\/a><\/li>\n<li><a href=\"#modifications\">Modifying Containers<\/a><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<h2><a name=\"import\"><\/a>Building From Docker or Singularity Hub<\/h2>\n<ol>\n<li style=\"list-style-type: none;\">\n<ol>\n<li>There are two dedicated nodes for building containers on the SCC. They are <code><b>scc-i01<\/b><\/code> and <code><b>scc-i02<\/b><\/code>. Login to either node with <code><b>ssh<\/b><\/code>.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc1 ~]<\/span> <span class=\"command\">ssh<\/span> <span class=\"placeholder\">scc-i01<\/span><\/code><\/pre>\n<\/li>\n<li>Run the following commands to create a directory on the local <code><b>\/scratch<\/b><\/code> drive for building the container. This significantly reduces the time it takes for the build:\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mkdir $TMPDIR\/$USER<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">SING_DIR=$TMPDIR\/$USER<\/span>\r\n<\/code><\/pre>\n<\/li>\n<li>Fetch the container <code><b>pull<\/b><\/code> command from Docker Hub (shown below), Singularity Hub, or other registry.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">singularity pull $SING_DIR\/<\/span><span class=\"placeholder\">CONTAINER_NAME.sif<\/span> <span class=\"command\">docker:\/\/<\/span><span class=\"placeholder\">PATH\/TO\/DOCKER\/CONTAINER<\/span><\/code><\/pre>\n<\/li>\n<li>Move the newly built container to your <code><b>\/projectnb<\/b><\/code> space.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mv $SING_DIR\/<\/span><span class=\"placeholder\">CONTAINER_NAME.sif<\/span> <span class=\"command\">\/projectnb\/<\/span><span class=\"placeholder\">PROJECT_NAME<\/span><\/code><\/pre>\n<\/li>\n<li>Delete the build directory when finished.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">rm -rf $SING_DIR<\/span><\/code><\/pre>\n<\/li>\n<li>Exit the build node.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">exit<\/span>\r\n<span class=\"prompt\">[rcs@scc1 ~]<\/span><\/code><\/pre>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<p>&nbsp;<\/p>\n<h2><a name=\"build\"><\/a>Building From Scratch<\/h2>\n<p>To build from a definitions (def) file on the SCC, you will need to do so from one of the dedicated build nodes to obtain <code><b>--fakeroot<\/b><\/code> privileges.<\/p>\n<ol>\n<li>Login to either build node (<code><b>scc-i01<\/b><\/code> or <code><b>scc-i02<\/b><\/code>) with <code><b>ssh<\/b><\/code>.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc1 ~]<\/span> <span class=\"command\">ssh<\/span> <span class=\"placeholder\">scc-i01<\/span><\/code><\/pre>\n<\/li>\n<li>Change to the build node <code><b>\/scratch<\/b><\/code> space and create a directory to build the container locally. The $USER variable just substitutes your username automatically.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mkdir $TMPDIR\/$USER<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">SING_DIR=$TMPDIR\/$USER<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">cd $SING_DIR<\/span>\r\n<\/code><\/pre>\n<\/li>\n<li>Build the container from the def file.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 $USER]<\/span> <span class=\"command\">singularity build --fakeroot<\/span> <span class=\"placeholder\">CONTAINER_NAME.sif CONTAINER_NAME.def<\/span><\/code><\/pre>\n<\/li>\n<li>Move the newly built container to your <code><b>\/projectnb<\/b><\/code> (or <code><b>\/rprojectnb<\/b><\/code>) space.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mv $SING_DIR\/<\/span><span class=\"placeholder\">CONTAINER_NAME.sif<\/span> <span class=\"command\">\/projectnb\/<\/span><span class=\"placeholder\">PROJECT_NAME<\/span><\/code><\/pre>\n<\/li>\n<li>Delete the build directory when finished.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span>  cd\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">rm -rf $SING_DIR<\/span><\/code><\/pre>\n<\/li>\n<li>Exit the build node.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">exit<\/span>\r\n<span class=\"prompt\">[rcs@scc1 ~]<\/span><\/code><\/pre>\n<\/li>\n<\/ol>\n<p>To see full instructions on designing a def file see <a href=\"https:\/\/docs.sylabs.io\/guides\/latest\/user-guide\/build_a_container.html#building-containers-from-singularityce-definition-files\">Sylab&#8217;s documentation<\/a>.<\/p>\n<h3>Common Problems<\/h3>\n<p>The most common problem when building a container from a .def file is an error stating there is &#8220;no space left on device.&#8221; The usual cause of this, assuming you are building the container on the <code><b>\/scratch<\/b><\/code>drive as recommended, is that the <code><b>\/tmp<\/b><\/code> filesystem has been filled. Singularity by default maps in the system <code><b>\/tmp<\/b><\/code> filesystem into the build process for temporary files, but that has a small (~2 GB) capacity. To work around this, add the following to your .def file before the %post section:<\/p>\n<pre><code class=\"code-block\">%setup\r\n    mkdir ${SINGULARITY_ROOTFS}\/scratch<\/code><\/pre>\n<p>Next, create a directory for temporary files on <code><b>\/scratch<\/b><\/code>, and use an environment variable to tell singularity to use that instead of <code><b>\/tmp<\/b><\/code>. When running the build command use the <code><b>-B \/scratch<\/b><\/code> flag to map that directory inside the container:<\/p>\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mkdir -p \/scratch\/$USER\/sing_tmpdir<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">export SINGULARITYENV_TMPDIR=\/scratch\/$USER\/sing_tmpdir<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">export SINGULARITY_TMPDIR=\/scratch\/$USER\/sing_tmpdir<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">singularity build --fakeroot -B \/scratch CONTAINER_NAME.sif CONTAINER_NAME.def<\/span><\/code><\/pre>\n<h2><a name=\"modifications\"><\/a>Modifying Containers<\/h2>\n<p>To modify your container, you will need to unpack the read-only image into a the writable sandbox. It is important that this is done on one of the dedicated build nodes to obtain <code><b>--fakeroot<\/b><\/code> privileges.<\/p>\n<ol>\n<li>Login to either build node (<code><b>scc-i01<\/b><\/code> or <code><b>scc-i02<\/b><\/code>) with <code><b>ssh<\/b><\/code>.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc1 ~]<\/span> <span class=\"command\">ssh<\/span> <span class=\"placeholder\">scc-i01<\/span><\/code><\/pre>\n<\/li>\n<li>Change to the build node <code><b>\/scratch<\/b><\/code> space and create a directory to unpack the container locally.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">mkdir $TMPDIR\/$USER<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">SING_DIR=$TMPDIR\/$USER<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">cd $SING_DIR<\/span>\r\n<\/code><\/pre>\n<\/li>\n<li>Unpack the container into a writable sandbox. Don&#8217;t use the <code><b>--fakeroot<\/b><\/code> while unpacking.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 $USER]<\/span> <span class=\"command\">singularity build --sandbox<\/span> <span class=\"placeholder\">CONTAINER_NAME_SANDBOX CONTAINER_NAME.sif<\/span><\/code><\/pre>\n<\/li>\n<li>Run the sandbox to make changes.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 $USER]<\/span> <span class=\"command\">singularity shell --fakeroot --writable<\/span> <span class=\"placeholder\">CONTAINER_NAME_SANDBOX<\/span>\r\n<span class=\"prompt\">Singularity&gt;<\/span><\/code><\/pre>\n<\/li>\n<li>When done modifying, pack the sandbox back into a read-only container.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 $USER]<\/span> <span class=\"command\">singularity build --fakeroot<\/span> <span class=\"placeholder\">MODIFIED_CONTAINER_NAME.sif CONTAINER_NAME_SANDBOX<\/span><\/code><\/pre>\n<\/li>\n<li>Move the modified read-only container back to your project space and delete the directory used.\n<pre><code class=\"code-block\"><span class=\"prompt\">[rcs@scc-i01 $USER]<\/span> <span class=\"command\">mv<\/span> <span class=\"placeholder\">MODIFIED_CONTAINER_NAME.sif<\/span> \/projectnb\/<span class=\"placeholder\">YOUR_PROJECT<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 $USER]<\/span>  cd\r\n<span class=\"command\"><span class=\"prompt\">[rcs@scc-i01 ~]<\/span> rm -rf $SING_DIR<\/span>\r\n<span class=\"prompt\">[rcs@scc-i01 ~]<\/span> <span class=\"command\">exit<\/span>\r\n<span class=\"prompt\">[rcs@scc1 ~]<\/span><\/code><\/pre>\n<\/li>\n<\/ol>\n<p>See more on building Singularity sandbox containers in the <a href=\"https:\/\/docs.sylabs.io\/guides\/3.0\/user-guide\/build_a_container.html#creating-writable-sandbox-directories\">Sylabs documentation<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sections Building From Docker or Singularity Hub Building From Scratch Modifying Containers &nbsp; Building From Docker or Singularity Hub There are two dedicated nodes for building containers on the SCC. They are scc-i01 and scc-i02. Login to either node with ssh. [rcs@scc1 ~] ssh scc-i01 Run the following commands to create a directory on the&#8230;<\/p>\n","protected":false},"author":1692,"featured_media":0,"parent":112635,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150138"}],"collection":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/users\/1692"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/comments?post=150138"}],"version-history":[{"count":25,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150138\/revisions"}],"predecessor-version":[{"id":156169,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150138\/revisions\/156169"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/112635"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/media?parent=150138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}