{"id":150684,"date":"2024-03-01T14:48:03","date_gmt":"2024-03-01T19:48:03","guid":{"rendered":"http:\/\/www.bu.edu\/tech\/?page_id=150684"},"modified":"2024-05-14T13:36:54","modified_gmt":"2024-05-14T17:36:54","slug":"large-memory","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/tech\/support\/research\/computing-resources\/external\/access\/large-memory\/","title":{"rendered":"ACCESS Large Memory Resources"},"content":{"rendered":"<p><a name=\"top\"><\/a><\/p>\n<style>\n  p { text-align: justify; }\n  .image-container { float: right; width: 44%; margin-left: 20px; }\n<\/style>\n<div style=\"width:100%;\">\n<div class=\"image-container\">\n    <img loading=\"lazy\" src=\"\/tech\/files\/2024\/04\/RP-bridge2.jpg\" alt=\"\" width=\"650\" height=\"400\" class=\"alignnone size-full wp-image-151447\" \/>\n  <\/div>\n<p>If your research regularly exceeds the memory limits of the SCC, we suggest allocating your project Service Units (SUs) on <a href=\"https:\/\/www.psc.edu\/resources\/bridges-2\/user-guide\/\">Bridges-2 EM<\/a>. After researching all available ACCESS resource options, we&#8217;ve found that the Bridges-2 extreme memory (EM) partition enables memory-intensive genome sequence assembly, graph analytics, in-memory databases, statistics, and other applications that need a large amount of memory and for which distributed-memory implementations are not available. You can explore other options on the <a href=\"https:\/\/allocations.access-ci.org\/resources\">ACCESS Resource list<\/a>.<\/p>\n<p>    <strong>Sections<\/strong><\/p>\n<ul>\n<li><a href=\"#bridges\">Bridges-2 EM<\/a><\/li>\n<li><a href=\"#start\">Getting Started on Bridges-2 EM<\/a><\/li>\n<li><a href=\"#help\">Bridges-2 EM Support<\/a><\/li>\n<li><a href=\"#example\">Example Large Memory Job on Bridges-2 EM<\/a><\/li>\n<\/ul>\n<p>    &nbsp;<\/p>\n<p>    <a name=\"bridges\"><\/a><\/p>\n<h2>Bridges-2 EM<\/h2>\n<p>Bridges-2 EM is a partition on the Bridges-2 cluster consisting of 4 extreme memory nodes funded by the NSF and administered by the Pittsburgh Supercomputing Center (<a href=\"https:\/\/www.psc.edu\/\">PSC<\/a>) at Carnegie Mellon University and the University of Pittsburgh.<\/p>\n<div style=\"margin-left: 20px;\">\n<h3>Architecture<\/h3>\n<p>Each extreme memory (EM) node:<\/p>\n<table>\n<tbody>\n<tr>\n<th>Architecture<\/th>\n<th>Cores<\/th>\n<th>RAM<\/th>\n<th>Cache<\/th>\n<th>Node-local storage<\/th>\n<th>Network<\/th>\n<\/tr>\n<tr>\n<td>Cascadelake<\/td>\n<td>96<\/td>\n<td>4TB<\/td>\n<td>37.75MB LLC x 6<\/td>\n<td>7.68TB NVMe<\/td>\n<td>Infiniband 200Gb\/s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Usage<\/h3>\n<p>The EM partition is only accessible via Bridges-2&#8217;s Slurm Workload Manager. They cannot be used for interactive nor OnDemand jobs. Memory is automatically allocated at 1TB per 24-cores and jobs must request 24, 48, 72, or 96-cores. As an example, if your job needs 2TB of memory, you should request 48 cores. The following table describes some of the resource bounds for using these EM nodes.<\/p>\n<table>\n<tbody>\n<tr>\n<th>Node Max<\/th>\n<th>Walltime Max<\/th>\n<th>Memory Max<\/th>\n<th>Cores Max<\/th>\n<th>Cores Min<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>120 hours<\/td>\n<td>4TB<\/td>\n<td>96<\/td>\n<td>24<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/div>\n<p>    &nbsp;<\/p>\n<p>    <a name=\"start\"><\/a><\/p>\n<h2>Getting Started on Bridges-2 EM<\/h2>\n<ol>\n<li>Once your Bridges-2 EM allocation is approved, you will receive a PSC account via email.<\/li>\n<li>Create a PSC password on the web-based <a href=\"https:\/\/apr.psc.edu\/\">PSC password change utility<\/a><\/li>\n<li>Connect to Bridges-2 via <b><code><span class=\"code\">ssh<\/span><\/code><\/b> to submit to the EM nodes:\n<pre><code class=\"code-block\"><span class=\"prompt\">[localhost$ ~] <\/span><span class=\"command\">ssh<\/span> <span class=\"placeholder\">myusername<\/span><span class=\"command\">@bridges2.psc.edu<\/span><\/code><\/pre>\n<\/li>\n<\/ol>\n<p>    &nbsp;<\/p>\n<p>    <a name=\"help\"><\/a><\/p>\n<h2>Bridges-2 Support<\/h2>\n<p>The help desk for Bridges-2 is PSC support. If you have questions at any time, you can send an email to <a href=\"mailto:help@psc.edu\">help@psc.edu<\/a>. They ask that you limit questions to one per email and provide the following:<\/p>\n<style>\n        .custom-list {\n            padding-left: 25px; \/* Add padding to the left for indentation *\/\n        }\n        .list-item {\n            position: relative; \/* Set position for pseudo-element *\/\n            margin-bottom: 5px; \/* Add some spacing between list items *\/\n        }\n        .list-item:before {\n            content: \"\u2022\"; \/* Use bullet character *\/\n            position: absolute; \/* Position the bullet *\/\n            left: -20px; \/* Adjust position of the bullet *\/\n            top: 0; \/* Align bullet with top of text *\/\n            font-size: 18px; \/* Adjust size of the bullet *\/\n        }\n        .indented-list {\n            margin-left: 25px; \/* Add margin to the left for indentation *\/\n    }\n    <\/style>\n<div class=\"indented-list\">\n<div class=\"custom-list\">\n<div class=\"list-item\">An informative subject line<\/div>\n<div class=\"list-item\">Your username<\/div>\n<div class=\"list-item\">If the question concerns a particular job, include these in addition:<\/p>\n<ul class=\"custom-list\" style=\"padding-left: 20px;\">\n<li>The JobID<\/li>\n<li>Any error messages you received<\/li>\n<li>The date and time the job ran<\/li>\n<li>Link to job scripts, output and data files<\/li>\n<li>The software being used, and versions when appropriate<\/li>\n<li>A screenshot of the error or the output file showing the error (if possible)<\/li>\n<\/ul><\/div>\n<\/p><\/div>\n<\/p><\/div>\n<p>    &nbsp;<\/p>\n<p>    <a name=\"example\"><\/a><\/p>\n<h2>Example Large Memory Job on Bridges-2 EM<\/h2>\n<p>Refer to our <a href=\"https:\/\/github.com\/bu-rcs\/access-use-case-demos\/tree\/main\/Large-Memory\">ACCESS GitHub page<\/a> for an example large memory job submission.<\/p>\n<p>    &nbsp;<\/p>\n<hr width=\"50%\" \/>\n    <i>This page was heavily inspired by the <a href=\"https:\/\/www.psc.edu\/resources\/bridges-2\/user-guide\/\">Bridges-2 User Guide at PSC<\/a>.<\/i><\/p>\n<\/div>\n<div style=\"float: right;\">\n<p id=\"last-modified-timestamp\" style=\"margin: 0;\">Last updated: Loading&#8230;<\/p>\n<\/p><\/div>\n<p><script>\r\ndocument.addEventListener('DOMContentLoaded', function() {\r\n    \/\/ Get the content of the meta tag\r\n    var lastUpdatedContent = document.querySelector('meta[name=\"last-updated\"]').content;\r\n    \r\n    \/\/ Parse the content into a Date object\r\n    var lastUpdatedDate = new Date(lastUpdatedContent);\r\n    \r\n    \/\/ Format the date\r\n    var formattedDate = lastUpdatedDate.toLocaleDateString(undefined, { year: 'numeric', month: 'long', day: 'numeric' });\r\n    \r\n    \/\/ Update the HTML element with the formatted date\r\n    document.getElementById('last-modified-timestamp').innerHTML = 'Last updated: ' + formattedDate;\r\n});\r\n<\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>If your research regularly exceeds the memory limits of the SCC, we suggest allocating your project Service Units (SUs) on Bridges-2 EM. After researching all available ACCESS resource options, we&#8217;ve found that the Bridges-2 extreme memory (EM) partition enables memory-intensive genome sequence assembly, graph analytics, in-memory databases, statistics, and other applications that need a large&#8230;<\/p>\n","protected":false},"author":1692,"featured_media":0,"parent":47915,"menu_order":5,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150684"}],"collection":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/users\/1692"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/comments?post=150684"}],"version-history":[{"count":22,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150684\/revisions"}],"predecessor-version":[{"id":152221,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/150684\/revisions\/152221"}],"up":[{"embeddable":true,"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/pages\/47915"}],"wp:attachment":[{"href":"https:\/\/www.bu.edu\/tech\/wp-json\/wp\/v2\/media?parent=150684"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}