How do I copy files between sites ?
Example: Download from BU’s SCC to Blacklight
kadin@tg-login1:~> mkdir MPI kadin@tg-login1:~> cd MPI kadin@tg-login1:~> scp firstname.lastname@example.org:/my_directory/MPI/lectures/MPIcodes11.zip . The authenticity of host 'scc1.bu.edu (220.127.116.11)' can't be established. RSA key fingerprint is a1:xx:yy:ed:zz:16:2f:1e:2b:c5:bb:0a:ac:r8:23:96. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'scc1.bu.edu,18.104.22.168' (RSA) to the list of known hosts. email@example.com's password: MPIcodes11.zip 100% 314KB 314.3KB/s 314.3KB/s 00:00
How do I unzip files ?
kadin@tg-login1:~/MPI> unzip MPIcodes11.zip Archive: MPIcodes11.zip creating: intro/ creating: intro/basics/ creating: intro/basics/C/ inflating: intro/basics/C/make.ch_gm.gnu inflating: intro/basics/C/make.ch_gm.intel . . . . . . . . . . kadin@tg-login1:~/MPI>
How do I compile MPI programs ?
SCF users beware: mpicc and mpif77 are not available on Blacklight. Use one of the two compilers (GNU and Intel) to compile the program, then link with the MPI library (-lmpi). Note that the GNU compilers are gcc and gfortran and the corresponding ones for Intel are icc and ifort. (More details …)
The following makefile demonstrates the compilation of serial C as well as C and C++ MPI codes.
##### Makefile ##### User configurable options ##### OPTFLAGS = -O3 CC = gcc $(OPTFLAGS) CCC = g++ $(OPTFLAGS) F77 = gfortran $(OPTFLAGS) F90 = gfortran $(OPTFLAGS) MAKE = make SHELL = /bin/sh # ### End User configurable options ### LIBS = -lm -lmpi -lmpi++ EXECS = example1 example1_1 example1_2 example1_3 ## all C++ suffixes work ## example1_1.cpp, example1_2.C, example1_3.cc default: $(EXECS) all: $(EXECS) example1: example1.o $(CC) -o $@ example1.o -lm example1_1: example1_1.o $(CCC) -o $@ example1_1.o $(LIBS) example1_2: example1_2.o $(CCC) -o $@ example1_2.o $(LIBS) example1_3: example1_3.o $(CCC) -o $@ example1_3.o $(LIBS) clean: /bin/rm -f *.o *.mod *~ fort.4* $(EXECS)
How to run jobs ?
You can run serial jobs interactively. Multiprocessor jobs must be submitted through batch. There are two queues: debug and batch. By default, jobs are submitted to the batch queue. Processor requests must be in increments of 16. Default wall time is 600 seconds. Current dir is not on the search path. Be sure to prepend the executable with “./”, like ./a.out. Examples:
Run serial job interactively
Interactive batch job
Multiprocessing jobs must run via batch (mpirun not available in login node). For program development and debugging, you can submit an interactive batch job, say, with 16 procs and 20 minutes duration. Once the batch job starts, you can proceed in the interactive mode.
~> qsub -I -l ncpus=16 -l walltime=20:00 ~> cd $HOME/MPI/intro/C ~/C> mpirun 3 ./example1_1 Process 0 has the partial integral of 0.500000 Process 1 has the partial integral of 0.366025 Process 2 has the partial integral of 0.133975 The Integral =1.000000
Submit a 16-proc batch job with a walltime of 5 minutes using the batch script my_batch_script
~> qsub -l ncpus=32 ./my_batch_script
my_batch_script looks like this: (the command-line request of 32 cores overrides the 16 cores predefined in the script)
#!/bin/csh #PBS -l ncpus=16 #ncpus must be a multiple of 16 #PBS -l walltime=5:00 #PBS -j oe #PBS -q batch # The purpose of the following is to demonstrate the use of the # scratch disk in the event that you expect large quantities of # temporary output that cannot be accomodated by your home dir. # Otherwise, there is no need to do so. #move to my $SCRATCH directory cd $SCRATCH #copy executable to $SCRATCH cp $HOME/MPI/intro/basics/C/example1_1 . #run my executable (in current dir) mpirun -np $PBS_NCPUS ./example1_1
You can use qstat to query the status of your job
qstat -u your-userid
The standard output goes to my_batch_script.oXXXXX.
I need 1 TB of memory for my job, what should I do ?
Memory is proportional to processor count. On Blacklight, each unit of 16 cores provides 128 GB. So, for 1 TB, you need to request 128 CPUs (-ncpus 128).
- gdb, idb – GNU and Intel debuggers, respectively (see man pages)
- tau – profiling tool, see Packages below
- PAPI – hardware performance counter software; see Packages below