For details, please consult the Blacklight User Guide.
For in depth understanding of the system, see this powerpoint presentation.

  • How do I copy files between sites ?

    Example: Download from BU’s SCC to Blacklight

    kadin@tg-login1:~> mkdir MPI
    kadin@tg-login1:~> cd MPI
    kadin@tg-login1:~> scp .
    The authenticity of host ' (' can't be established.
    RSA key fingerprint is a1:xx:yy:ed:zz:16:2f:1e:2b:c5:bb:0a:ac:r8:23:96.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added ',' (RSA) to the list
    of known hosts.'s password:                  100%  314KB 314.3KB/s 314.3KB/s   00:00
  • How do I unzip files ?

    kadin@tg-login1:~/MPI> unzip
       creating: intro/
       creating: intro/basics/
       creating: intro/basics/C/
      inflating: intro/basics/C/make.ch_gm.gnu
      inflating: intro/basics/C/
       . . . . .
       . . . . .
  • How do I compile MPI programs ?

    SCF users beware: mpicc and mpif77 are not available on Blacklight. Use one of the two compilers (GNU and Intel) to compile the program, then link with the MPI library (-lmpi). Note that the GNU compilers are gcc and gfortran and the corresponding ones for Intel are icc and ifort. (More details …)

    The following makefile demonstrates the compilation of serial C as well as C and C++ MPI codes.

    ##### Makefile
    ##### User configurable options #####
    OPTFLAGS    = -O3
    CC          = gcc $(OPTFLAGS)
    CCC         = g++ $(OPTFLAGS)
    F77         = gfortran $(OPTFLAGS)
    F90         = gfortran $(OPTFLAGS)
    MAKE        = make
    SHELL       = /bin/sh
    ### End User configurable options ###
    LIBS = -lm -lmpi -lmpi++
    EXECS = example1 example1_1 example1_2 example1_3
    ## all C++ suffixes work
    ## example1_1.cpp, example1_2.C,
    default: $(EXECS)
    all: $(EXECS)
    example1: example1.o
            $(CC) -o $@ example1.o -lm
    example1_1: example1_1.o
            $(CCC) -o $@ example1_1.o $(LIBS)
    example1_2: example1_2.o
            $(CCC) -o $@ example1_2.o $(LIBS)
    example1_3: example1_3.o
            $(CCC) -o $@ example1_3.o $(LIBS)
            /bin/rm -f *.o *.mod *~ fort.4* $(EXECS)
  • How to run jobs ?

    You can run serial jobs interactively. Multiprocessor jobs must be submitted through batch. There are two queues: debug and batch. By default, jobs are submitted to the batch queue. Processor requests must be in increments of 16. Default wall time is 600 seconds. Current dir is not on the search path. Be sure to prepend the executable with “./”, like ./a.out. Examples:

    • Run serial job interactively

      ~> ./a.out

    • Interactive batch job

      Multiprocessing jobs must run via batch (mpirun not available in login node). For program development and debugging, you can submit an interactive batch job, say, with 16 procs and 20 minutes duration. Once the batch job starts, you can proceed in the interactive mode.

      ~> qsub -I -l ncpus=16 -l walltime=20:00
      ~> cd $HOME/MPI/intro/C
      ~/C> mpirun 3 ./example1_1
      Process 0 has the partial integral of 0.500000
      Process 1 has the partial integral of 0.366025
      Process 2 has the partial integral of 0.133975
      The Integral =1.000000
    • Submit a 16-proc batch job with a walltime of 5 minutes using the batch script my_batch_script

      ~> qsub -l ncpus=32 ./my_batch_script

      my_batch_script looks like this: (the command-line request of 32 cores overrides the 16 cores predefined in the script)

      #PBS -l ncpus=16
      #ncpus must be a multiple of 16
      #PBS -l walltime=5:00
      #PBS -j oe
      #PBS -q batch
      # The purpose of the following is to demonstrate the use of the
      # scratch disk in the event that you expect large quantities of 
      # temporary output that cannot be accomodated by your home dir.
      # Otherwise, there is no need to do so.
      #move to my $SCRATCH directory
      cd $SCRATCH
      #copy executable to $SCRATCH
      cp $HOME/MPI/intro/basics/C/example1_1 .
      #run my executable (in current dir)
      mpirun -np $PBS_NCPUS ./example1_1

      You can use qstat to query the status of your job

      ~> qstat -u your-userid

      The standard output goes to my_batch_script.oXXXXX.

  • I need 1 TB of memory for my job, what should I do ?

    Memory is proportional to processor count. On Blacklight, each unit of 16 cores provides 128 GB. So, for 1 TB, you need to request 128 CPUs (-ncpus 128).

  • Programming Tools

    • gdb, idb – GNU and Intel debuggers, respectively (see man pages)
    • tau – profiling tool, see Packages below
    • PAPI – hardware performance counter software; see Packages below
  • Application Packages