If your research regularly exceeds the memory limits of the SCC, we suggest allocating your project Service Units (SUs) on Bridges-2 EM. After researching all available ACCESS resource options, we’ve found that the Bridges-2 extreme memory (EM) partition enables memory-intensive genome sequence assembly, graph analytics, in-memory databases, statistics, and other applications that need a large amount of memory and for which distributed-memory implementations are not available. You can explore other options on the ACCESS Resource list.

Sections

 

Bridges-2 EM

Bridges-2 EM is a partition on the Bridges-2 cluster consisting of 4 extreme memory nodes funded by the NSF and administered by the Pittsburgh Supercomputing Center (PSC) at Carnegie Mellon University and the University of Pittsburgh.

Architecture

Each extreme memory (EM) node:

Architecture Cores RAM Cache Node-local storage Network
Cascadelake 96 4TB 37.75MB LLC x 6 7.68TB NVMe Infiniband 200Gb/s

Usage

The EM partition is only accessible via Bridges-2’s Slurm Workload Manager. They cannot be used for interactive nor OnDemand jobs. Memory is automatically allocated at 1TB per 24-cores and jobs must request 24, 48, 72, or 96-cores. As an example, if your job needs 2TB of memory, you should request 48 cores. The following table describes some of the resource bounds for using these EM nodes.

Node Max Walltime Max Memory Max Cores Max Cores Min
1 120 hours 4TB 96 24

 

Getting Started on Bridges-2 EM

  1. Once your Bridges-2 EM allocation is approved, you will receive a PSC account via email.
  2. Create a PSC password on the web-based PSC password change utility
  3. Connect to Bridges-2 via ssh to submit to the EM nodes: 
    [localhost$ ~] ssh myusername@bridges2.psc.edu

 

Bridges-2 Support

The help desk for Bridges-2 is PSC support. If you have questions at any time, you can send an email to help@psc.edu. They ask that you limit questions to one per email and provide the following:

An informative subject line
Your username
If the question concerns a particular job, include these in addition:

  • The JobID
  • Any error messages you received
  • The date and time the job ran
  • Link to job scripts, output and data files
  • The software being used, and versions when appropriate
  • A screenshot of the error or the output file showing the error (if possible)

 

Example Large Memory Job on Bridges-2 EM

Refer to our ACCESS GitHub page for an example large memory job submission.

 

This page was heavily inspired by the Bridges-2 User Guide at PSC.

Last updated: Loading…