General Usage of parpool

Invoking parpool submits a batch job to start a parallel environment.

>> parpool(n)

where n is the number of labs (i.e. workers). If, in addition, spmd is invoked, then this parallel environment is almost equivalent to pmode but without the interactive GUI layout. By invoking spmd, labindex and numlabs, as are available in pmode, are now available. If spmd is not enabled (as in a parfor application), labindex is not needed and is not available. However, there are times when you might need to know the number of processors, numlabs. In that situation, you can do the following:

>>p = gcp
>>n = p.NumWorkers
n = 4
  • parfor can be used with parpool (but not inside spmd). If parpool(n) is not explicitly executed before, the parfor will automatically launch as many workers as possible; that is, the number of workers will be equal to the maximum number of physical cores on the node.
  • >> parfor i=1:4, disp(['myid is ' num2str(labindex) '; i = ' num2str(i)]),end  # labindex is not enabled in parfor
myid is 1; i = 4
myid is 1; i = 3
myid is 1; i = 2
myid is 1; i = 1
>> for j=1:4, disp(['myid is ' num2str(labindex) '; j = ' num2str(j)]),end
myid is 1; j = 1
myid is 1; j = 2
myid is 1; j = 3
myid is 1; j = 4
>> x = 0; parfor i=1:10, x = x + i; end   % reduction operation
>> y = []; parfor i=1:10, y = [y, i]; end   % concatenation
>> z = []; parfor i=1:10, z(i) = i; end    % slicing
>> f = zeros(1,50); f(1) =  1; f(2) = 2;
>> parfor n=3:50
f(n) = f(n-1) + f(n-2);
end % Fibonacci numbers; not parallelizable because of data dependency. The op fails.
% Next is a reduction; but "-" violates associative rule, and the op fails
>> u = 0; parfor i=1:10, u = i - u; end
  • spmd
>> spmd x = labindex; disp(['for myid = ' num2str(labindex) '; x = ' num2str(x)]),end

Lab 1:
  for myid = 1; x = 1
Lab 2:
  for myid = 2; x = 2
Lab 3:
  for myid = 3; x = 3
Lab 4:
  for myid = 4; x = 4

    The first column of output above are the rank numbers printed automatically by spmd. Without activating spmd, you have no access to labindex (or numlabs).

  • >> delete(gcp) % to close all pre-existing or dangling parpool jobs
  • A nondistributed array is a shared array. Any change made to the array by a lab causes the change to be seen by all labs.
  • A replicated array (nondistributed) resides on the MATLAB client. This array is visible from the labs.
  • A variant array resides, in its entirety, in an individual lab's workspace. This is a composite array. A composite array can communicate between the MATLAB client and labs directly. A composite array can be manipulated from the client. In this case, the {labnumber} must always be used. Here is an example: 
    >> A = Composite();
>> A = magic(4);   % A is a replicate array
>> A{2} = ones(4);   % Change A on lab 2 directly from MATLAB client
  • A codistributed array is a single array divided into segments (e.g., columns of a 2D array), each residing in the workspace of a lab.
  • Codistributed arrays
    Array of this type are partitioned into segments, each residing in a different lab. This results in memory saving, ideal for large arrays.

    >> spmd
A = [11:18; 21:28; 31:38; 41:48]
D = codistributed(A)
end
>> spmd
D
size(D)    % this will report size of the global array
L = getLocalPart(D)    % L is the same as D, locally
size(L)    % this gives the size of the local array
end
  • non-distributed arrays
    Arrays created on the MATLAB client, before or after parpool, are nondistributed arrays (replicated and variant arrays); entire array is stored on each lab.

Codistributed arrays

  • Why create a codistributed array?
    For more efficient parallel computing and memory usage.
  • How to create a codistributed array?
    • Partitioning a larger array: 
      >> parpool(4)
>> spmd
A = [11:18;21:28;31:38;41:48];   % replicate array
D = codistributed(A)  % codistributed
end
    • Building from smaller arrays: 
      >> parpool(4)
>> spmd
A = [11:13; 21:23; 31:33; 41:43] + (labindex-1)*3;  % variant array
D = codistributed(A);
end
    • Using MATLAB constructor functions: 
      >> D = zeros(1000, codistributor());
    • codistributed is normally used in the parallel environment, like spmd and pmode.
    • codistributed is similar to the scatter function in MPI.
    • gather is the opposite of "codistributed".
    • numel is supported for codistributed arrays. It returns the size of the codistributed arrays.
  • Indexing into a Codistributed Array: 
    >> spmd
A = [11:18; 21:28; 31:38; 41:48];   % A is a replicated array
D = codistributed(A);    % D is A distributed on labs; saves memory
L = getLocalPart(D);                   % L is a local copy of D
L(3,:)   % prints row 3 of local array
n = size(L,2);  % column size of L
L(3,1:n) % same as above
D(3,:)   % PCT treats D as if localPart(D)
D(3,1:end) % same as above
C = getCodistributor(D)  % obtain info of the codistributed array
s = C.Partition   % size of local partitions
last = sum(s(1:labindex));       % global ending element index for local worker
first = last - s(labindex) + 1;  % global beginning element index
D(3,first:end) % first:end is global indexing; D is hence global
D(3,1:n)       % explicit indexing causes D to be treated as global;
               % fails on labs 2 to 4
end

Configuring parpool for Batch Submission

In order to use parpool in a batch job, two things need to be configured: (1) the number of assigned cores to avoid the process reaper and (2) a temporary directory for log files to avoid simultaneous parpools from writing to the same directory. To do this, include the following code in your script replacing pipeline with the name of your script and parameter with some unique input available to your script such as your job ID:


% Set up 'local' parpool cluster
pc = parcluster('local')

% Discover number of available cores assigned by SGE
nCores = str2num(getenv('NSLOTS'))

% Setup directory for temporary parpool files
parpool_tmpdir = ['~/.matlab/local_cluster_jobs/pipeline/pipeline_' parameter ]
mkdir(parpool_tmpdir)
pc.JobStorageLocation = parpool_tmpdir

% Start parpool
parpool(pc,nCores)