Scientific Computing Facilities Technical Summary
Table of contents
The charge rate in “service units” (SUs) for each system is given below or you can see this table listing all of the charge rates together.
Listed below are the technical details on our systems, including run time limits and queue structures. If for some reason your code is not able to run within these parameters, please don’t hesitate to send e-mail to help@scv.bu.edu to let us know of any problems or difficulties you have in using our systems. We will do everything in our power to help you with your issue.
Katana Cluster
Hardware Configuration
| Host Name(s) & Node Type |
# of Nodes | Processors / Node | Cache | Memory (per node) |
Scratch Disk | Network | SU Charge for each CPU hour |
|---|---|---|---|---|---|---|---|
| katana.bu.edu (Login Node) |
1 | 2 dual-core 2.6 GHz AMD Opteron 2218HE | 64 KB L1 and 1 MB L2 |
8 GB | 46 GB | 1 Gbps Ethernet SDR Infiniband |
1.0 |
| katana-a02..a14, katana-b01..b08 4 Processor, 2.6 Ghz Compute Nodes |
21 | 2 dual-core 2.6 GHz AMD Opteron 2218HE | 64 KB L1 and 1 MB L2 |
8 GB | 46 GB | 1 Gbps Ethernet SDR Infiniband |
1.0 |
| katana-b09..b14 8 Processor, 3.0 Ghz Compute Nodes |
6 | 2 quad-core 3.0 GHz Intel Xeon E5450 | 32 KB L1 and 12 MB L2 |
16 GB | 113 GB | 1 Gbps Ethernet SDR Infiniband |
1.5 |
| katana-c01..c14 4 Processor, 2.4 Ghz Compute Nodes |
14 | 2 dual-core 2.4 GHz AMD Opteron 2216HE | 64 KB L1 and 1 MB L2 |
8 GB | 46 GB | 1 Gbps Ethernet | 0.9 |
| katana-d01..d08 8 Processor, 2.93 Ghz Compute Nodes |
8 | 2 quad-core 2.93 GHz Intel Xeon X5570 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
24 GB | 251 GB | 1 Gbps Ethernet QDR Infiniband |
1.9 |
| katana-d091,geo1 8 Processor, 2.93 Ghz Compute Nodes |
2 | 2 quad-core 2.93 GHz Intel Xeon X5570 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
24 GB | 251 GB | 1 Gbps Ethernet QDR Infiniband |
1.9/0.01 |
| katana-d11..d12, 4 Processor, 2.2 Ghz Compute Nodes |
2 | 2 dual-core 2.2 GHz AMD Opteron 275 | 64 KB L1 and 1 MB L2 |
4 GB | 31 GB | 1 Gbps Ethernet | 0.0 |
| katana-d13..d14 8 Processor, 2.93 Ghz Compute Nodes1 |
2 | 2 quad-core 2.93 GHz Intel Xeon X5570 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
96 GB1 | 251 GB | 1 Gbps Ethernet QDR Infiniband |
2.4 |
| katana-e01..e031 12 Processor, 2.93 Ghz Compute Nodes |
3 | 2 six-core 2.93 GHz Intel Xeon X5670 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
48 GB | 499 GB | 1 Gbps Ethernet QDR Infiniband |
1.9/0.01 |
| katana-e04..e131 12 Processor, 2.93 Ghz Compute Nodes |
10 | 2 six-core 2.93 GHz Intel Xeon X5670 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
96 GB | 499 GB | 1 Gbps Ethernet | 1.9/0.01 |
| katana-f01..f14, katana-g01..g14 4 Processor, 2.4 Ghz Compute Nodes |
28 | 2 dual-core 2.4 GHz AMD Opteron 2216HE | 64 KB L1 and 1 MB L2 |
8 GB | 46 GB | 1 Gbps Ethernet | 0.9 |
| katana-h01..h02 12 Processor, 2.93 Ghz Compute Nodes |
2 | 2 six-core 2.93 GHz Intel Xeon X5670 | 32 KB Instruction 32 KB Data 256 KB mid-level 8MB L2 Shared |
48 GB | 133 GB | 1 Gbps Ethernet | 1.9 |
1These machines have limited access; not all SCF users can fully utilize these systems. For those users with special access to these systems, the SU charge is 0.0 for these systems only.
Note on large memory nodes: These nodes will only be used if the -l memory=96G option to qsub is specified. See the Running Jobs page for more details.
Batch System and Usage
The batch system on the Katana Cluster is the Sun Grid Engine.
A user can submit as many jobs as desired. However, no more than 64 processors of a single user can be in the run state simultaneously. The maximum run time limit is generally 24 hours (-l h_rt=24:00:00). However, we are now allowing a limited number of jobs per user to run up to 72 hours. A user can request up to 4 processors (as 4 single processor jobs or one 4 processor job for example) with a run time limit of 72 hours; we currently have 12 slots for this purpose among all users. The default limit is 2 hours if you do not specify a higher limit.
Depending on the speed, memory, and other factors, each node type is charged at a different SU rate per hour, as shown in the table above. The SU charges for all of ours systems is also detailed in this table and varies on the Katana cluster from 0.0 SUs per CPU hour to 2.4 SUs per CPU hour.
Blue Gene
Hardware Configuration
Each node contains two processors so the number of processors is twice the number of nodes listed.
| Machine Name | Role in Cluster | # Nodes | Cache | Memory (per node) | Scratch Disk | Network |
|---|---|---|---|---|---|---|
| Levi and Lee | Login Node | 2 x 1.5 GHz Power5 | see below | 4 GB | 72 GB | 1Gbps Ethernet |
| Compute Nodes | 1024 x 700 MHz PPC440 | see below | 512 MB | none | Torus, Tree, Global Interrupt | |
| IO Nodes | 128 x 700 MHz PPC440 | see below | 512 MB | none | 1 Gbps Ethernet, Tree |
Note on front end caches: Each processor has a 64 KB instruction, 32 KB data L1 cache. Each node has a shared 1.9MB L2 and 36 MB L3 cache.
Note on compute and IO node caches: Each node has a 32 KB L1 cache, 2 KB L2 cache, and a 4 MB L3 cache.
Batch System and Usage
The IBM Blue Gene’s batch system is IBM’s LoadLeveler. The current limitation is that all jobs must use a partition of exactly 32, 128, 512 or 1024 (the entire machine) nodes and no job may run for more than 5 hours of wall-clock time. 1024-node jobs are only allowed to run in off-hours.
Each CPU hour (counted by wall clock time) used on the Blue Gene will use 0.25 SUs of a project’s allocation. Note that on all of our other non-Blue Gene systems, usage is charged by actual CPU usage rather than by wall clock time.
pSeries
Hardware Configuration
| Host Name | Model | # Processors | Cache | Memory (Aggregate) | Scratch Disk | Network |
|---|---|---|---|---|---|---|
| Twister | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1 Gbps Ethernet |
| Scrabble | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1 Gbps Ethernet |
| Marbles | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1 Gbps Ethernet |
| Crayon | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1 Gbps Ethernet |
| Litebrite | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1 Gbps Ethernet |
| Hotwheels | IBM pSeries 655 | 8 x 1.1 GHz Power4 | see below | 16 GB | 36 GB | 1Gbps Ethernet |
| Jacks | IBM pSeries 655 | 8 x 1.7 GHz Power4 | see below | 8 GB | 72 GB | 1 Gbps Ethernet |
| Playdoh | IBM pSeries 655 | 8 x 1.7 GHz Power4 | see below | 8 GB | 72 GB | 1 Gbps Ethernet |
| Slinky | IBM pSeries 655 | 8 x 1.7 GHz Power4 | see below | 8 GB | 72 GB | 1 Gbps Ethernet |
Note on 655 caches: 32 KB L1 cache on each processor, 1.41 MB L2 cache shared by each pair of processors, 128 MB L3 cache shared by each set of eight processors.
Batch System and Usage
The IBM pSeries machines have a detailed queue structure and certain machines dedicated to certain job types as indicated below. The batch system used to manage the queues is the Load Sharing Facility (LSF).
The charge rate for the 1.1 GHz nodes is 0.85 SUs per CPU hour. For the faster 1.7 GHz nodes (jacks, playdoh, and slinky), the charge rate is 1.31 SUs per CPU hour.
| Host Name | Function | Service Level | Batch Queues |
|---|---|---|---|
| Twister | Interactive | Production | none |
| Scrabble | SP batch MP batch |
Production | p4-short p4-verylong p4-mp4 |
| Marbles | SP batch | Production | p4-long |
| Crayon | SP batch | Production | p4-long |
| Litebrite | MP batch | Production | p4-mp8 |
| Hotwheels | MP batch | Production | p4-mp8 |
| Jacks | MP batch/Interactive1 | Production | p4-cism-mp8 |
| Playdoh | MP batch/Interactive1 | Production | p4-cism-mp8 |
| Slinky | MP batch/Interactive1 | Production | p4-cism-mp8 |
[Notes: SP=Single Processor; MP=Multiple Processor]
| Queue Name | # processors | CPU Limit (in hours) |
Wall Clock Limit (in hours) |
Run Window | Slots |
|---|---|---|---|---|---|
| p4-short | 1 | 2 | 2.5 | always | 2 |
| p4-long | 1 | 32 | 40 | always | 16 |
| p4-verylong | 1 | 64 | 80 | always | 2 |
| p4-mp4 | 4 | 16 | 5 | always | 1 |
| p4-mp8 | 8 | 32 | 5 | always | 2 |
| p4-cism-mp8 | 8 | 32 | 5 | always | 3 |
Note: In general, all of the LSF queues will have dedicated access to all of the processors they may access (“# processors” column above) as none of the machines are allowed to be oversubscribed.
Computer Graphics Lab Workstations
The Computer Graphics Lab houses a number of high performance Linux and Windows workstations. The lab is accessible only to those with appropriate card keys. Follow the above link for more details on the machines available and getting access to the lab.