Running MATLAB on HPC Clusters

Introduction

A tutorial on running serial and parallel MATLAB jobs on CPUs and GPUs on CCAST clusters.

MATLAB is a high-level programming language and numerical computing environment developed by MathWorks. It is used for numerical computation, machine learning, and visualization. MATLAB is available to all CCAST users who are affiliated with North Dakota State University. There are multiple MATLAB versions installed on CCAST systems. On Thunder or Thunder Prime, check all available software modules by typing:

$ module avail matlab

There are two ways to run MATLAB: one uses batch mode on Thunder or Thunder Prime, and the other one is via Open OnDemand. That latter allows you to run MATLAB interactively and graphically. In this document, we only discuss running serial and parallel MATLAB jobs on CPUs and GPUs via batch mode.

A MATLAB job needs MATLAB scripts that you intend to run and a job submission script to submit it to the job scheduler (PBS Pro on Thunder or Thunder Prime). See the CCAST User Guide for more information on running jobs on CCAST systems in general.

Example files

All the source codes and job submission scripts discussed in this document can be found in the following compressed file: /mmfs1/thunder/projects/ccastest/examples/MATLAB_Tutorial_examples.tar.gz (on Thunder)
/mmfs1/projects/ccastest/examples/MATLAB_Tutorial_examples.tar.gz (on Thunder Prime)

To copy the examples to your SCRATCH directory:

# on Thunder
$ cp /mmfs1/thunder/projects/ccastest/examples/MATLAB_Tutorial_examples.tar.gz $SCRATCH
# on Thunder Prime
$ cp /mmfs1/projects/ccastest/examples/MATLAB_Tutorial_examples.tar.gz $SCRATCH

Then, extract the files:

$ cd $SCRATCH
$ tar -xvf MATLAB_Tutorial_examples.tar.gz

Running serial jobs on CPUs

In the following, we examine a few MATLAB examples:

Example 1: Hello World !

In this simple example, the system prints the phrase “Hello World !” in the output file.

$ cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_1_HelloWorld

The MATLAB script: HelloWorld.m

% display information
fprintf(Hello World !\n\n)

The job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q default
#PBS -N test
## serial jobs: only 1 processor core is requested
#PBS -l select=1:mem=2gb:ncpus=1
#PBS -l walltime=00:10:00
## replace 'x-ccast-prj' below with 'x-ccast-prj-[your project group name]'
#PBS -W group_list=x-ccast-prj

## if you require a specific version of MATLAB, replace "module load matlab" below
module load matlab

cd $PBS_O_WORKDIR

## change the input filename as needed
matlab -nodesktop -nodisplay -r "run HelloWorld.m"

exit 0

Note that the above job submission script requires the group_list variable to be set to your project group name.

To submit the job:

$ qsub matlab_job.pbs

To check the status of all jobs you are running (It may show nothing if the job has been completed):

$ qstat -u $USER

The job will produce an output file named test.o<job_id>, where <job_id> is the job ID number. The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020
To get started, type doc.
For product information, visit www.mathworks.com.

Hello World !

Example 2: Summing the elements of a vector

In this example, the job creates a vector outside of a loop and uses the vector index to do the sum operation of a vector. Then, the system prints the sum value in the output file.

cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_2_Sum

The MATLAB script: sum.m

z = 0;
xAll = 0:0.1:10000; % create a vector outside of a loop
for i= 1:length(xAll)
    x = xAll(i);
    z = z + x;
end
fprintf('%f\n', z);
% Copyright 2010 - 2014 The MathWorks, Inc.

The job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q default
#PBS -N test
## serial jobs: only 1 processor core is requested
#PBS -l select=1:mem=2gb:ncpus=1
#PBS -l walltime=00:10:00
## replace "x-ccast-prj" below with "x-ccast-prj-[your project group name]"
#PBS -W group_list=x-ccast-prj

## if you require a specific version of MATLAB, replace "module load matlab" below
module load matlab

cd $PBS_O_WORKDIR

## change the input filename as needed
matlab -nodesktop -nodisplay -r "run sum.m"

exit 0

The above script requests a single node for at most 10 minutes, with 2 GB of RAM. After editing the job submission script as needed (editing the line “#PBS -W”), to submit this job:

$ qsub matlab_job.pbs

The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020

To get started, type doc.
For product information, visit www.mathworks.com.
500005000.000000

Running parallel jobs on CPUs

MATLAB parallelism includes implicit multi-threaded parallelism and explicit parallelism. In this section, we will discuss how to run parallel jobs on CPUs.

Implicit parallelism

MATLAB contains internal mechanisms that enable some code to run much faster by automatically parallelizing arithmetic and logical operations on data. This is called implicit parallelism (multi-threaded or built-in multi-threading) since we do not need to explicitly tell MATLAB to parallelize the operations. Implicit parallelization relies on the fact many operations are independent of each other and can therefore be processed in parallel.

For implicit parallel computation, vector operation is the necessary trigger, but it is not sufficient. The application or algorithm, and the amount of computation also help MATLAB to determine whether an application will be performed with multi-threads. This link provides more information on conditions of implicit parallelism. Such parallelism is implemented by multi-threaded computations that are executed within a single node. Therefore, on Thunder or Thunder Prime, you should only select one node to enable implicit parallelism.

Example 3: Matrix Multiplication

This job executes multiplication operations by multiplying a matrix by another matrix. In this example, implicit parallelism is triggered. This can be observed by log in to the compute node and check the CPU usage. On Thunder or Thunder Prime, to check the name of the compute node, you can execute:

$ cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_3_Multiplication

The MATLAB script: matrix_multiplication.m

n = 5000; % set matrix size
A = rand(n); % create random matrix
B = rand(n); % create another random matrix
tic % calculate the elapsed time using tic and toc
C = A \* B; % matrix multiplication
time=toc; % calculate the elapsed time using tic and toc
disp(['Processing time: ' num2str(time)] 's'); % display running time (unit: second)

The job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q default
#PBS -N test
##change the ncpus number to run the job in implicit parallel
#PBS -l select=1:mem=4gb:ncpus=1
#PBS -l walltime=00:30:00
##replace "x-ccast-prj" below with "x-ccast-prj-[your project group name]"
#PBS -W group_list=x-ccast-prj

## to run on Thunder Prime, replace "module load matlab/R2020a" below with "module load matlab/R2021a"
module load matlab/R2020a

cd $PBS_O_WORKDIR

##change the input filename as needed
matlab -nodesktop -nodisplay -r "run matrix_multiplication.m"

exit 0

To submit this job:

$ qsub matlab_job.pbs

The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020

To get started, type doc.
For product information, visit www.mathworks.com.

Processing time: 6.8422s

To observe implicit parallel computing performance, when different numbers of cores are selected, you might get results like:

The results show the relationship between the processing time and the number of cores allocated. The running time will not decrease monotonously with the increase of the number of cores. Users need to decide the number of selected cores according to the actual situation of the project.

Explicit parallelism

Explicit parallelism is characterized by the presence of explicit constructs in the programming language, aimed at describing the way in which the parallel computation will take place. In explicit parallelism, several instances of MATLAB simultaneously execute a single MATLAB command or function.

The Parallel Computing Toolbox describes the explicit parallelism and lets you use parallel-enabled functions in MATLAB and other toolboxes. For more details on running your code in explicit parallelism, see Choose a Parallel Computing Solution.

A common way to initiate a parallel computation in MATLAB is to use a parfor loop: parfor executes for-loop iterations in parallel on workers in a parallel pool. For more information, please see Decide When to Use parfor.

Example 4: Getting eigenvalues of square matrices

This job performs N trials of computing the largest eigenvalue for a M-by-M random matrix using parfor and outputs its processing time.

$ cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_4_MatrixEigenvalue

The MATLAB script: parfor.m

% Performs N trials of computing the largest eigenvalue for an M-by-M random matrix
gcp; % open parallel pool if none is open
M=500; % number of rows and columns of each matrix
N=12000; % number of trials
tic; % calculate the elapsed time using tic and toc
a = zeros(N,1); % initialize output vector
parfor i = 1:N % use parallel processing by running parfor in a parallel pool
a(i) = max(eig(rand(M))); % vector of largest eigenvalues
end
time = toc; % calculate the elapsed time using tic and toc
disp(['Parallel processing time: ' num2str(time)] 's'); % display running time
poolobj = gcp('nocreate'); % Get current parallel pool
delete(poolobj); % Shutting down parallel pool

The job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q default
#PBS -N test
#PBS -l select=1:mem=8gb:ncpus=5
#PBS -l walltime=00:30:00
##replace "x-ccast-prj" below with "x-ccast-prj-[your project group name]"
#PBS -W group_list=x-ccast-prj

##to run on Thunder Prime, replace "module load matlab/R2020a" below with "module load matlab/R2021a"
module load matlab/R2020a

cd $PBS_O_WORKDIR

##change the input filename as needed
matlab -nodesktop -nosplash -nodisplay -r "run parfor.m"

exit 0

To submit this job:

$ qsub matlab_job.pbs

The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020

To get started, type doc.
For product information, visit www.mathworks.com.

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 5).
Parallel processing time: 33.9835s
Parallel pool using the 'local' profile is shutting down.

To observe explicit parallel computing performance, a different number of cores are selected, and you might get results like:

Example 5: for-loop and parfor-loop

$ cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_5_for_CompareLoops

This example job also performs N trials of computing the largest eigenvalue for an M-by-M random matrix, but it addresses another two problems. First, this job compares processing time between for-loop processing and parfor-loop processing. Second, when your code includes files that call other files, you need to use addpath to add the working directory into your MATLAB search path.

The MATLAB script: ex_compare.m

% set pathToData with your current work directory
% pathToData = [CURRENT_WORKDIR];
addpath(pathToData);
tic; a1 = ex_for(500,1200); t1 = toc; % calculate a1 using for-loop
gcp; % open parallel pool if none is open
tic; a2 = ex_parfor (500,1200); t2 = toc; % calculate a2 using parfor-loop
% Compare processing times
disp(['For-loop processing time: ' num2str(t1) 's'])
disp(['Parfor-loop processing time: ' num2str(t2) 's'])
% Shutting down parallel pool
poolobj = gcp('nocreate'); % get current parallel pool
delete(poolobj); % shutting down parallel pool
% Copyright 2010 - 2014 The MathWorks, Inc.

Which calls two functions: ex_parfor.m and ex_for.m

ex_parfor.m

function a = ex_parfor(M, N)
a = zeros(N,1);
parfor i = 1:N
vi a(i) = max(eig(rand(M)));
end

ex_for.m

function a = ex_for(M, N)
a = zeros(N,1);
for i = 1:N
a(i) = max(eig(rand(M)));
end

Job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q default
#PBS -N test
#PBS -l select=1:mem=8gb:ncpus=2
#PBS -l walltime=00:30:00
##replace "x-ccast-prj" below with "x-ccast-prj-[your project group name]"
#PBS -W group_list=x-ccast-prj

##to run on Thunder Prime, replace "module load matlab/R2020a" below with "module load matlab/R2021a"
module load matlab/R2020a

cd $PBS_O_WORKDIR

##pass $PBS_O_WORKDIR to MATLAB to add directories into MATLAB search path
sed -i "1c pathToData = '$PBS_O_WORKDIR';" compare.m

##change the input filename as needed
matlab -nodesktop -nosplash -nodisplay -r "run compare.m"

exit 0

To submit this job:

$ qsub matlab_job.pbs

The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020

To get started, type doc.
For product information, visit www.mathworks.com.

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 2).
For-loop processing time: 141.6204s
Parfor-loop processing time: 72.8218s
Parallel pool using the 'local' profile is shutting down.

This job compares processing time between for-loop and parfor-loop under the same workload. The results show that parfor-loop processing time is much less than for-loop processing time.

Running MATLAB on GPUs

A GPU, designed to quickly render high-resolution images and video concurrently, and can also be used for non-graphical tasks such as machine learning and scientific computation due to its ability to perform parallel operations on multiple sets of data. MATLAB provides tools to run functions on a GPU, including gpuArray for transferring input data and calling gather to retrieve output data from the GPU. Many functions in MATLAB and other toolboxes are automatically enabled on a GPU if they support GPU-enabled computing, including fast Fourier transform fft, matrix multiplication mtimes, left matrix division mldivide, and hundreds of others. For more information, please see Accelerate Your Code by Running It on a GPU.

Example 6: Filter a matrix

$ cd $SCRATCH/MATLAB_Tutorial_examples/MATLAB_example_6_GPU_MatrixFilter

This job filters a matrix on GPU using the filter function and returns the processing time on GPU.

The MATLAB script: matrix_filter.m

A = magic(5000); % magic(n) returns an n-by-n matrix
f = ones(1,20)/20; % create array of all ones
% Filter a matrix on GPU
AonGPU = gpuArray(A); % create an array stored on the GPU
tic; % calculate the elapsed time using tic and toc
BonGPU = filter(f, 1, AonGPU); % do filter operation on GPU
C=gather(BonGPU); % convert back to a numeric array on the CPU
wait(gpuDevice) % wait for GPU calculation to complete
tCompGpu = toc; % calculate the elapsed time using tic and toc
disp([' Processing time on GPU: ' num2str(tCompGpu) 's']) % display processing time
% Copyright 2014 The MathWorks, Inc.

The job submission script: matlab_job.pbs

#!/bin/bash
#PBS -q gpus
#PBS -N test
#PBS -l select=1:ncpus=1:mem=4gb:ngpus=1
#PBS -l walltime=00:20:00
##change “x-ccast-prj” to “x-ccast-prj-[your project group name]"
#PBS -W group_list=x-ccast-prj

##to run on Thunder Prime, replace "module load matlab/R2020a" below with "module load matlab/R2021a"
module load matlab/R2020a

cd $PBS_O_WORKDIR

##change the input filename as needed
matlab -nodesktop -nodisplay -r "run matrix_filter.m"

exit 0

To submit this job:

$ qsub matlab_job.pbs

The expected output is:

                            < M A T L A B (R) >
                  Copyright 1984-2020 The MathWorks, Inc.
              R2020a Update 3 (9.8.0.1396136) 64-bit (glnxa64)
                                May 27, 2020

To get started, type doc.
For product information, visit www.mathworks.com.
Processing time on GPU: 3.9936s

From the output, you can see the computation time of filter function on GPU.

For more information on parallel computing, see the MATLAB website.