Programs are scheduled to run on Tiger using the sbatch
command, a component of Slurm. Your job will be put into the appropriate quality of service, based on the requirements that you describe. See Usage Guidelines
and the sbatch
man page for details.
The Gnu and Intel compilers are installed on Tiger. The standard MPI implementation for Tiger is OpenMPI, an MPICH compatible library that supports the Infiniband infrastructure.
To set up your environment correctly on Tiger, it is highly recommended to use the module
facility. This is a utility to correctly set your environment without having to know all the paths to the executables. Different environments can be set quickly allowing useful comparisons of code compiled with different executables. In most cases a simple module load openmpi
command can be issued setting up your environment to use the latest openmpi as well as the Intel compilers.
Compiling parallel MPI programs
module load openmpi (loads the openmpi environment as well as the Intel compilers)
Submitting a Job
Once the executable is compiled, a job script will need to be created for the scheduler. For this machine there are a total of 16 processor cores per node. Special node properties include the large memory nodes (mem128) and GP-GPUs (gpu). These nodes can be chosen by adding those special flags to the requirements (nodes=2:ppn=16:mem128 for example).
Here is a sample script which uses 32 processors allocated as 16 processors on 2 nodes:
# parallel job using 32 processors. and runs for 4 hours (max)
#SBATCH -N 2
#SBATCH -t 4:00:00
# sends mail when process begins, and
# when it ends. Make sure you define your email
#SBATCH --mail-user= yourNetID@princeton.edu
module load openmpi
mpiexec ./a.out (Change "NetID" to your NetID in the above script.)
To submit the job to the batch queuing system use:
To allocate a job using the GPUs (up to 4 per node) you will need to add another specifier to the #SBATCH command requesting the amount of GPUs as well as sending it to the GPU partition. Because of the way that the GPU's are laid out on a node, you must allocate both sockets if you require more than 2 GPUs.
#SBATCH -N 1