The Research Computing Services (RCS) group at Boston University maintains the Shared Computing Cluster (SCC) to provide high performance computing to the BU research community. On occasion, a research project may have special needs that are not adequately served by the SCC. For those projects, the external resources provided through the XSEDE program may be appropriate.
XSEDE (eXtreme Science and Engineering Discovery Environment) is a virtual system that provides compute resources for scientists and researchers from all over the country. Its mission is to facilitate research collaboration among institutions, enhance research productivity, provide remote data transfer, and enable remote instrumentation. XSEDE is funded by National Science Foundation (NSF).
Here is the Getting Started Guide for XSEDE.
Training and workshops on various topics are available throughout the year.
What compute resources are available?
XSEDE compute resources are administrated by several universities and institutions across the country. These compute resources include various powerful computer clusters for high-performance computing (HPC), high throughput computing (HTC) and cloud computing. In particular, XSEDE provides the following kinds of resources that are not available on the BU SCC (Please see the technical summary of SCC).
- An extremely large number of CPU cores.
Currently the maximum number of CPU cores for a single MPI job on SCC is 256. However, you can run an MPI job with thousands of CPU cores on XSEDE clusters. If you have an MPI program that speeds up perfectly with the number of CPU cores, it is a good idea to migrate it to XSEDE clusters.
- Large-memory and extremely-large-memory nodes.
The size of memory on large-memory nodes of SCC is 256 GB, while there are large-memory nodes on many XSEDE clusters providing up to 1 TB memory per node and extremely-large-memory nodes on the Bridges cluster providing up to 3 TB memory per node. If you need to run a program that consumes more than 256 GB memory, it is recommended to use XSEDE resources instead.
- A large number of advanced GPUs.
There are 160 GPUs on SCC. XSEDE provides a larger number of GPUs with newest architectures (such as K40 and K80). If you want to use more GPUs or to explore new GPU architectures, you might consider using XSEDE resources.
- Xeon-Phi (MIC) coprocessors/processors.
There is no Xeon-Phi coprocessor on SCC currently. Several XSEDE clusters (such as Stampede and SuperMIC) provide a lot of first-generation Xeon-Phi coprocessors (Knights Corner). The second-generation Xeon-Phi processor (Knights Landing) will be available to public users soon. If you want to use Xeon-Phi coprocessor/processor to accelerate your program, XSEDE resources is a good option.
- Advanced storage system.
Parallel I/O is not supported by SCC currently, while most of the XSEDE clusters support parallel I/O with advanced storage system (such as Lustre). Some XSEDE clusters (such as Gordon) provide solid-state drives (SSD) particularly for I/O bound applications. If your program requires parallel I/O or a very high I/O speed, XSEDE provides you appropriate resources.
- High throughput computing system
The Open Science Grid supports high throughput computing and it is available to XSEDE users.
- Cloud computing system
A new XSEDE system, Jetstream, provides cloud computing services.
- An extremely large number of CPU cores.
How do I get an XSEDE User Portal account?
In order to use XSEDE resources, you must have an XSEDE User Portal (XUP) account. You can create an XUP account at the XSEDE user portal home page.
How do I apply for XSEDE allocations ?
Here is an overview of XSEDE allocations. There are three primary considerations:
- Who qualifies?
The Principle Investigator (PI) of an XSEDE project must be a faculty, staff, post-doc of a US-based educational institution or member of a commercial organization. Members of the project may be any researcher, including graduate students, post-docs and visiting scholars. For details, see XSEDE PI qualifications.
- Which type of computer systems should I apply for?
To determine the system that best match your hardware and software requirements, please see available resources.
- Which type of allocations (CPU time and disk storage allotments)?
There are 4 types of allocations (BU RCS , startup, education and research) available for BU XSEDE users. A summary is given below.
BU RCS allocations
This is the simplest and quickest way to “sample” resources.
If you are not sure which resources are appropriate for your computing needs, this is the right type of allocation to start with. BU, as an XSEDE member institution, has been given allocations by a few of the resource providers (institutions) across the country through XSEDE. RCS staffs can add you as a member under this project, subject to approval by XSEDE.
PROS: no need to write a proposal; time to approval is two business days.
CONS: you can only run relatively short duration jobs and are limited to the resources available to Boston University.
Resource Provider Machine
Key Features Status Gordon San Diego Supercomputing Center (SDSC). Host name:
Cluster Good for IO-bound apps, SSD (FLASH) storage. Util 2016-08-31 Open Science Grid Over 100 individual sites spanning the United States. Host name: grid1.osg.xsede.org Virtual Cluster For distributed high throughput computing (DHTC). Util 2017-05-30 Maverick Texas Advanced Computing Center (TACC).
Host name: maverick.tacc.xsede.org
Cluster For visualization, VNC server, GPUs, large memory. Production Stampede TACC. Host name: stampede.tacc.xsede.org Cluster Good for massive MPI jobs, the largest cluster in XSEDE, a large number of Intel Xeon Phi (MIC) coprocessors, GPU, big memory nodes. Until 2017-01-05 SuperMIC Louisiana State University (LSU). Host name: supermic.cct-lsu.xsede.org Cluster Good for massive MPI jobs, Intel Xeon Phi (MIC) coprocessors, big memory nodes. Production
Startup, education and research allocations
The startup allocation is suitable for researchers who need a moderate amount of resources or as a first step towards a more significant research allocation. You may apply at any time. It requires a one-page abstract. Approval process is 2 weeks. Allocation expires after one year. Then a startup-allocation PI is encouraged to proceed with a follow-up research allocation request. However, renewal of an startup allocation will be permitted with appropriate justification and subject to XSEDE reviewer approval.
The education allocation enables the PI to provide classroom instruction or training activities. The policy for an education allocation is similar to that for a startup allocation.
The research allocation enables the PI to continue research they began with startup allocations. However, a startup allocation is not a prerequisite for requesting a research allocation. A successful research allocation request requires a detailed justification of resource usage (a 10 to 15 page proposal). Requests are reviewed quarterly by the XSEDE Resource Allocations Committee. Since it is competitive to apply for research allocations, PIs need to prepare a strong proposal. It is recommended to watch an online training for Writing and Submitting a Successful XSEDE Allocation Proposal.
For details on up-to-date policies, please consult the XSEDE allocation policy page.
Once you have an XUP account, you can access to the XSEDE Resource Allocation System (XRAS) to submit applications for these three types of allocations.
- Who qualifies?
How do I login?
There are several ways to log in to an XSEDE system on which you have an allocation. The easiest way is to log in via the Single Sign On (SSO) login hub. Alternatively, there is the GSI-SSHTerm method for logging in from your personal computer. If you prefer to use direct ssh, you will probably need to apply for a userid and password for the specific system on which you have an allocation.
- Single Sign On (SSO) Login Hub
The Single Sign On login hub (login.xsede.org) is a single point-of-entry to the XSEDE resources. Upon logging into the hub with your XUP userid and password, a 12-hour proxy certificate is automatically generated, allowing the user to access XSEDE resources for the duration of the proxy. Users may then gsissh to any XSEDE system without the need for a system-specific userid and password.
PROS: The advantage is that there is a universal userid/password for all the XSEDE systems you have access to.
First, download and install the GSI-SSHTerm. With this software, you can connect to an XSEDE host system from a PC (running Windows XP or Window 7), Mac, or Linux with the XUP userid and password.
- Direct ssh
Most sites, e.g., LSU SuperMIC, TACC stampede and maverick, require a separate userid and password to access via direct
ssh. You can submit a ticket to the XSEDE helpdesk (see the last paragraph of this page) requesting for the direct-ssh access.
- Single Sign On (SSO) Login Hub
How do I transfer files?
You may need to transfer files between an XSEDE system and your personal computer or between two XSEDE systems. There are two ways to transfer files: scp/sftp or Globus. Please refer to XSEDE Data Transfers & Management.
- scp/sftp: All of the XSEDE resources are Linux/Unix clusters, so you can use scp or sftp to transfer files. Please refer to the SCC user guide for scp and sftp.
- Globus: Globus is based on the grid-ftp protocol. If you want to transfer a large amount of data (e.g. in the scale of terabytes), it is better to use Globus to obtain a faster transfer speed. For usage of Globus, please refer to Globus User Guide for XSEDE users.
Other information ?
- Since a lot of jobs are continually running on XSEDE resources, it is possible that your job will be waiting in the queue. You can use the wait time prediction tool to estimate how soon your job will start, so as to optimize your job submission.
- We will have BU RCS allocations on the following new XSEDE resources for the next cycle (starting on Jun 2016).
Jetstream at Indiana University (IU) and TACC: a cloud computing resource;
XStream at Stanford University (SU): with a large number of GPUs;
Bridges at Pittsburgh Supercomputing Center (PSC): with extremely-large-memory nodes.
- A supercomputer, Anton, built specifically for molecular dynamics (MD) simulations is open to academic institutions, including BU.
- Shaohao Chen (firstname.lastname@example.org, 617-353-8294) and Keith Ma (email@example.com, 617-358-1873) are the XSEDE liaisons at Boston University. Contact them for general questions such as allocations, programming, and HPC issues. Contact the XSEDE helpdesk (see below) for help on system-specific issues.
- If you want to ask for help from a specific XSEDE site, you can submit a ticket to XSEDE helpdesk or send email to firstname.lastname@example.org . They also have a 24/7 phone service (1-866-907-2383).