Computational Cluster Programs

Running an Array of Jobs Using SGE on an ATS-Hosted Cluster

SGE Job Arrays

The Sun Grid Engine (SGE) has a job array option, which lets you submit multiple jobs with a single command file. It is convenient to use job arrays when you need to conduct a parametric study to optimize the various parameters in an experiment. Computing the optimum solution involves repeated execution of same program over a range of input values and this is exactly what the job array option lets you do.

The UCLA Grid Portal has a feature called Multi-Job that is very similar to the Sun Grid Engine job array option. If you are using the UCLA Grid Portal to submit jobs, you can accomplish the same thing by using that service. The concepts are the same so the examples shown here do apply. Only the names of the environment variables differ.

To run using job arrays, you need to insert an instruction of the form:

#$ -t lower-upper:interval

into your SGE command file.

The -t option defines the task index range, where lower is the smallest index, upper the upper index bound and interval the interval to use. SGE runs your executable once for each number in the task index range: lower, lower+interval, lower+2*interval, ..., upper and it generates a variable, $SGE_TASK_ID, for each instance which your program can use to determine its task number for that run. Your program should use the task number to select the input files to read or the options to use for that particular run.

Here is an example, which demonstrates the use of the job array facility.

Script Example:

File: hellotask.sh

#!/bin/sh
echo "Task id is $SGE_TASK_ID"
if [ -e $HOME/JOBARRAY/data$SGE_TASK_ID.in ]; then
while read file
do
echo "hello $file"
done < $HOME/JOBARRAY/data$SGE_TASK_ID.in
fi

File: data1.in

first
second

File: data2.in

third
fourth

When the value of $SGE_TASK_ID is 1, the hellotask.sh script will output:

Task id is 1
hello first
hello second

When it is 2, the output will look like:

Task id is 2
hello third
hello fourth

C++ Program Example:

This example just prints the value of the $SGE_TASK_ID environment variable.

#include <iostream>
#include <sstream>
#include <string>

int
     main( int argc, char *argv[ ] )
{
     std::stringstream testID;

     testID << getenv( "SGE_TASK_ID" );
     std::cerr << " Test ID is " << testID.str( ) << std::endl;

     return 0;
}

Using the Queue Script to Submit a Job Array

If you are not using the UCLA Grid Portal, we encourage you to use the queue scripts to to build, submit and monitor batch jobs rather than writing SGE command files yourself. This eliminates mistakes. Use the jobarray.queue script to create a job array command file. When you run the job array, this will create a separate job log file and output file for each instance of the executable that Job Arrays runs for you.

To run the a program with jobarray.queue:

• Modify the program to be run to read an input file whose name include a sequence number. If the program is a script, use the $SGE_TASK_ID variable in place of the number in the file name as shown in the script example above. If the program is in a compiled language, use the getenv( ) function, as shown in the C program example above and compile the program into an executable.


• Save any input files the program reads in the same directory as the program (executable or script) and name them with names that include the sequence numbers.


• Run the jobarray.queue script to create a command file to submit the job to SGE. The script will ask you for the name of the script or executable to run and also for the lower index, upper index, interval etc. Then it will create a command file for you. The output and log files created by this command file will have the SGE_TASK_ID appended to their names.


• Submit the command file either using the jobarray.queue script or the qsub command.