Computational Cluster Programs

How to Run MPI

MPI Versions available on ATS-Hosted Clusters

On ATS-hosted clusters we use the following implementations of MPI: MPICH and MPICH2 from Argonne National Lab for Ethernet, MVAPICH1 and MVAPICH2 from The Ohio State University for Infiband and, the MPI from Myricom, for Myrinet.

Network	MPI 1	MPI 2
Ethernet	MPICH1	MPICH2, Open MPI
InfiniBand	MVAPICH1	MVAPICH2, Open MPI
Myrinet	Myricom

Cluster	Networks	MPI Implementations	Compilers	Defaults
Hoffman2	Infinband Ethernet	MVAPICH2, Open MPI MPICH2	Intel	Open MPI and Intel
Cardio	Myrinet	Myricom	Intel	Myricom and Intel
CNSI2	Ethernet	MPICH1	PathScale	MPICH1 and PathScale
Hydro	Infinband Ethernet	MVAPICH1 and MVAPICH2 MPICH1 and MPICH2	Intel	MVAPICH2 and Intel
Neutrino	Ethernet	MPICH1	Intel	MPICH1 and Intel

MPI-1 is deprecated and MPI-1 implemtations, including MPICH1, are no longer under development.

Selecting an MPI Version, Interconnect, and Compiler other than the Default

The default combination for a cluster has been selected to give you the best performance with the hardware/software available on that cluster. Only set your path as described in this section to use an MPI version/compiler combination other than the default.

To use a version of MPI/compiler combination other than the default, add the appropriate compiler path as shown in the table below IN FRONT of your PATH environment. For the bash shell:

export PATH=pathToAdd:$PATH

For the tcsh shell:

setenv PATH pathToAdd:${PATH}

You can only select an MPI/compiler combination that is available on the cluster you are using. See the table above.

MPI Implementation Compiler	Add this path to the front of your PATH
MVAPICH1 Intel PathScale Portland Group Nag	/u/local/mpi/mvapich1/current/bin /u/local/mpi/pathscale/mvapich1/current/bin /u/local/mpi/pg/mvapich1/current/bin /u/local/mpi/nag/mvapich1/current/bin
MVAPICH2 Intel PathScale Portland Group Nag	/u/local/mpi/mvapich2/current/bin /u/local/mpi/pathscale/mvapich1/current/bin /u/local/mpi/pg/mvapich1/current/bin /u/local/mpi/nag/mvapich1/current/bin
MPICH1 Intel PathScale Portland Group Nag	/u/local/mpi/mpich1/current/bin /u/local/mpi/pathscale/mpich1/current/bin /u/local/mpi/pg/mpich1/current/bin /u/local/mpi/nag/mpich1/current/bin
MPICH2 Intel PathScale Portland Group Nag	/u/local/mpi/mpich2/current/bin /u/local/mpi/pathscale/mpich2/current/bin /u/local/mpi/pg/mpich2/current/bin /u/local/mpi/nag/mpich2/current/bin
Open MPI Intel	/u/local/mpi/openmpi/current/bin

Compiling and Linking MPI Programs

The following commands are used to compile/link mpi programs. The commands are the same no matter what MPI version, interconnect, and compiler you are using. If you have not modified your path, as described above, you will get the default MPI version, interconnect and compiler for the cluster you are using. Otherwise, what you get will be determined by how you have set your path.

Language	Command Used to Compile
Fortran 77	mpif77
Fortran 90	mpif90
C	mpicc
C++	mpiCC

Examples:

mpif90 -o myprog myprog.f90

Compiles myprog.f90 and creates the myprog executable.

mpicc -c myprog.c

Compiles myprog.c and creates the myprog.o object file.

mpiCC -o myprog myprog.o

Links the C++ object file myprog.o with the appropriate MPI libraries and creates the myprog executable.

Running an MPI Program as a Batch Job

There are three ways to submit an MPI batch job to the Hoffman2 Cluster. They are from easiest to hardest:

from the UCLA Grid Portal
via the queue script:
mpi.q
by generating an SGE command file for the job and using the SGE commands to submit it.

Instructions are given in Running a Batch Job on an ATS-Hosted Cluster.

Running an MPI Program Interactivly for Testing

NOTE: Not all clusters have interactive nodes. You can only debug an MPI program interactively on clusters that do.

While you can DEBUG or TEST a parallel program for a short time on the interactive nodes of an ATS-hosted cluster, you cannot run a full run on the interactive nodes. After testing, a parallel program must be submitted to the SGE batch queuing system to run on the compute nodes.

To test a parallel program ssh to one of the interactive nodes, and follow the instructions for the version of MPI you are using.

MPI 2 (using MPICH2 or MVAPICH 2)

Create a host file with one line for each process to be run excluding the local host where you are currently logged on. For example, on the Hoffman2 Cluster the interactive nodes are named i01 and i02 and you are logged into i01. You could create a host file named hostfile with the following commands:
echo "i01:4" > hostfile
echo "i02:4" >> hostfile

Where 4 is due to the fact that i01 and i02 both have 4 processors. Then issue the following commands to run the program:
mpdboot --ncpus=4 -n 2 -f hostfile
mpiexec -n 8 pathToExectutable
Replace hostfile with the name of your hostfile and pathToExecutable with the name of your executable (relative or full path).

After your program has completed enter:
mpdallexit
to clean up and release the hosts.

MPI 2 (using Open MPI)

Create a host file with one line for each process to be run excluding the local host where you are currently logged on. For example, on the Hoffman2 Cluster the interactive nodes are named i01 and i02 and you are logged into i01. You could create a host file named hostfile with the following commands:
echo "i01 slots=4" > hostfile
echo "i02 slots=4" >> hostfile

Where 4 is due to the fact that i01 and i02 both have 4 processors. Then issue the following commands to run the program:
mpirun -machinefile hostfile -np 8 pathToExecutable < input > output

MPI 1

Create a host file indentically to the way you would for MPI 2. Then issue the command:
mpirun -machinefile hostfile -np 8 pathToExecutable < input > output

Replace hostfile with the name of your hostfile, pathToExecutable with the name of your executable (relative or full path), and input and output with the names of your standard input and output files.

After your program completes, you MUST enter:
cleanupmpinodes
to kill any orphan MPI processes that may have inadvertently been left behind. This is especially important if any of your processes terminated abnormally or did not run to completion normally.