Research Computing

    module load mpt/intel

    mpif90 myMPIcode.f
    mpicc myMPIcode.c

Once an executable is compiled, a job script needs to be created for PBS. Here is a sample command script, parallel.cmd, which uses 12 processors. Note that though Hecate is a shared memory system, the processors are organized into cpu sets, where each virtual node has 6 processors. So processor allocations must use the same nodes and ppn keywords used on other clusters, and not the ncpus specification used by the previous Hecate. Also, the ppn keyword must always be set to 6, so as to allocate an entire virtual node. Finally, the mem keyword must be set to the max number of GBs needed by the entire application.

The script is located in mpi_directory, a subdirectory of the user's home directory.

cd <my_mpi_directory>
cat parallel.cmd

#!/bin/bash
# parallel job using 12 processors, a max of 96 GB memory, and runs for a max of 4 hours.
#PBS -l nodes=2:ppn=6,mem=96g,walltime=4:00:00
#
# sends mail if the process aborts, when it begins, and
# when it ends (abe). Make sure you define your email
# address as it will just get sent to the local machine
# without this.
#
#PBS -m abe
#PBS -M netid@princeton.edu

module load mpt/intel

cd /home/netid/mpi_directory
mpiexec_mpt ./a.out

To submit the job to the batch system, use:
qsub parallel.cmd

While it is possible to build and run jobs using OpenMPI instead of MPT, this is not recommended on Hecate. Performance using OpenMPI is significantly worse than with MPT.