Computational Cluster Programs

Using OpenMP

OpenMP is a parallel programming specification used to create C/C++ and Fortran shared-memory parallel programs on multi-processor nodes. For large problems, where you want to run on more processors than are available on a single node, you can combine MPI with OpenMP to run on more than one node, where some or all of the nodes are SMP nodes. OpenMP is portable across a number of platforms and compilers. If you are unfamiliar with the OpenMP Standard and would like additional information to that which is provided here, you can read the OpenMP Specifications.

OpenMP Directives

With OpenMP, parallel execution on shared memory processors is achieved by adding parallelization directives to existing programs. The form of a parallelization directive for Fortran is:

!$OMP   directive

and for C/C++ is:

#pragma omp  directive

Complier Flags

To enable OpenMP add the following compiler flag when you compile your program:

Compiler	Flag
Intel	-openmp
Pathscale	-openmp
Portland Group	-mp
GNU	-f openmp

OpenMP Environment Variables

OMP_NUM_THREADS

To run an OpenMP job in parallel you MUST set the OMP_NUM_THREADS runtime environment variable to specify the number of threads to use during the execution of the parallel regions of the code. If you omit setting this variable, it will default to 1 and your program will not run in parallel. While you can request a number of threads that is greater than the number of physical processors (cores) on the node and the program will execute correctly, the performance of programs executed in this manner is unpredictable and is often poor. Therefore, DO NOT specify a number of threads that is greater than the number of cores per node on the cluster you are using.

For the C Shell specify:

setenv OMP_NUM_THREADS number

For the Bash Shell specify:

export OMP_NUM_THREADS=number

Other OpenMP Environment Variables

The following table lists other, less often used, environment variables that can be set at runtime.

Variable	Description	Default
OMP_SCHEDULE	Sets the run-time schedule type and chunk size.	static, no chunk size specified
OMP_NESTED	Enables or disables nested parallelism.	false
OMP_DYNAMIC	Enables or disables the dynamic adjustment of the number of threads.	false

Running an OpenMP Program as a Batch Job

There are two ways to submit an OpenMP batch job to the Hoffman2 Cluster. They are from easiest to hardest:

• via the queue script:

  mpishm.q

• by generating an SGE command file for the job and using the SGE commands to submit it.

Instructions are given in Running a Batch Job on an ATS-Hosted Cluster. The UCLA Grid Portal cannot be used to submit OpenMP jobs.