Enabling Multi-level Parallelism in Parallel MATLAB

Supervisor

Suitable for

Abstract

Parallel MATLAB has been proved to be an effective platform for achieving context free parallelism. Although MATLAB cannot compete with main-stream parallel programming models such as MPI, OpenMP or CUDA, it is certainly an easier route for performing parallel computation without being exposed to any intricacies of parallel computing. The ability to utilise an existing set of computational abstractions in MATLAB is an additional advantage. Parallel MATLAB can easily make use of multiple cores, multiple CPUs and multiple nodes within a cluster for parallelising a MATLAB code. Parallel MATLAB relies on two main constructs, parfor and spmd for achieving parallelism. However, MATLAB does not have the notion of nested parallelism. In other words, MATLAB views all computational resources at one level. The aim of this project is to enhance the existing setting so that we could realise the multi-level parallelism in MATLAB .

1. Introduction
MATLAB is a very powerful platformfor developing and prototyping simulation models. The key advantage ofMATLAB
is the abstractions provided therein. However, MATLAB applications are compiled at runtime and then run— a technology
known as just-in-time (JIT) compilation (or jitting). This approach renders the MATLAB applications to run very slowly but
enables specialisations and special optimisations. However, MATLAB JIT does not make use of any parallel computational
resources, such as multiple cores or CPUs. Parallel MATLAB [2] is aimed to rectify this condition by using a pool of
resources such as multi-core CPUs and clusters.
Parallel MATLAB achieves the parallelism by running several instances, called workers, on a closely federated set of resources,
such as cluster of nodes or number of cores in a single node. For example, in a given cluster of N nodes each with C
cores, MATLAB would see N × C computational cores. MATLAB can be configured to subscribe one or more workers per
core and thus parallelMATLAB can instantiate a number of workers at any given time. This resource set can be shared across
a number of users: one user may allocate n1 workers, while another may allocate n2 workers where n1 + n2 = kNC. Each
allocation is known as a matlabpool . Given a matlabpool , parallel computations organised through the parallel
MATLAB constructs like parfor and spmd cannot cross the pool boundaries— meaning that they cannot use any more
than allocated workers. Furthermore, workers are allocated without differentiating remoteness or locality. This means that
when allocating a set of workers, they may span more than one node or may be allocated within the same node.
Internally, this matlabpool is a Message Passing Interface (MPI) ring [1]. MATLAB takes care of passing, relaying and
sharing any data necessary for parallel computation. This notion of message passing shields the notion of locality. Although
this notion brings the advantage of making use of all available resources and a unified representation of programs for all
hardware models, it also has a number of drawbacks such as:
• Workers allocated within the same node will share the memory. This is very detrimental in memory intensive simulations,
particularly when making heavy use of FFT kernels. Since the subscription mechanism is not dynamic and it is
a global policy, it is not possible to demand this at runtime.
• Given a pool, all workers are connected through a ring. This renders forming grouped but yet parallel computations a
much harder option. For instance, in solving a certain class of parallel problems, it may be necessary to formulate a
number of groups each with a set of workers. Here, the parallelism may exist within and beyond the group level.
These two drawbacks contribute to various issues including difficulty in supporting nested parfor or nested spmd blocks.
The purpose of this project is to enable MATLAB to support dynamic grouping and dynamic nested parallelism.

2 Dynamic Grouping and Nested Parallelism
MATLAB provides a basic set of MPI-level constructs to coordinate with different workers. By appropriately extending/
augmenting these constructs, it is possible to formulate parallel computing groups in MATLAB. This, when combined
with the rest of the parallel MATLAB framework, should deliver a robust mechanism for efficiently addressing different
classes of computational problems.
The ability to control the worker dispatch at runtime provides a better control over the behaviour of MATLAB programs.
Although such a fine control may not reach all user groups, it is an essential mechanism to enable MATLAB to handle
complex problems.

3 Components of the Project
The project consists of well balanced parts of research and development. The research component will expect the candidate
to follow up any existing work aligned with the motivation outlined in this project. In some cases, it may be necessary to
extend or to develop novel ideas and algorithms.

4 Workload and Work Schedule
This document outlined two different aspects. Although it is beneficial to address both the issues, it is understood that
addressing the dynamic worker dispatch mechanism is a difficult aspect. The key aim is to augment capability of the parallel
MATLAB to address complex problems and to check the hypothesis that the approach could bring potential performance
gains within MATLAB.
In cases where more than one candidate opts for this project, or when this project is awarded as a group project (maximum 2
members), the work load can be shared.

Schedule
1. Read key papers and manuals including that of ParallelMATLAB,MPI-specific programming aspects and parallel and
grouped computations.
2. Familiarising with MPI-level programming within MATLAB. This will involve using existing MATLAB constructs
for writing relatively simple yet real-world MATLAB applications.
3. Identify and extend/augment the basic set of MATLAB MPI constructs to cover missing MPI constructs. For example,
this may include constructing MPI-BROADCAST, MPI-ALL-TO-ALL and alike.
4. Introduce and develop the notion of groups in MATLAB. This will include introducing new scripts to MATLAB.
5. Develop/port MATLAB applications making use of grouped computations using the grouping constructs introduced
above.
6. Evaluate the performance gains/losses using a suite of applications.
7. Investigate different mechanisms for introducing dynamic, user-controlled worker dispatch within MATLAB.

5 Candidate
Desire to complete the project and moderate exposure to MATLAB. Knowledge of MPI will be an added advantage, but not
necessary.

6 Contact
Jeyan Thiyagalingam and Anne Trefethen, OeRC
Email: jeyarajan.thiyagalingam@oerc.ox.ac.uk

For more information, please contact Jeyan Thiyagalingam, OeRC
Email: jeyarajan.thiyagalingam@oerc.ox.ac.uk