Research Computing

The head nodes, sesame1, sesame2, and sesame3, should be used for interactive work only, such as compiling programs, and submitting jobs as described below. No jobs should be run on the head node, other than brief tests that last no more than a few minutes.

All jobs must be run through the scheduler on Sesame. This cluster uses a number of preemptive queues to control access and provide resources in a timely fashion to those who have contributed the most.  Because of this, the default queue, risk, has the highest probablity of having a job be interrupted.  The other queues, dedicated for specific groups, all have higher priority. Users belonging to the astro, burrows, caf, eac, hep, medvigy, and tromp groups should specify their queue in the job submission script. To submit to a specific queue, use a PBS directive such as:

 

If this line is not present, jobs will move to the risk queue.  Only users in the individual groups should use these special queues.

 

Jobs are further prioritized through the maui scheduler based on a number of factors: job size, run times, node availability, wait times, and percentage of usage over a 7 day period.

 

There are 3584 processors available, 8 per node. Each node contains 24GB of memory. The nodes are identified as r1c1n1 through r7c4n16 -where there are 7 racks containing 4 chassis each with 16 nodes per chassis.

 

The nodes are all connected through eight QDR Infiniband switches for MPI traffic and over a Gigabit Ethernet for NFS and Lustre I/O and normal communication.

 

/home (shared via NFS to all the compute nodes, 1.2 TB) is intended for scripts, source code, executables and small static data sets that may be needed as standard input/configuration for codes.

 

/scratch/network (shared via NFS to all the compute nodes, 1.3 TB) is intended for dynamic data that doesn't require high bandwidth i/o such as storing final output for a compute job. You may a create a directory/scratch/network/myusername, and use this to place your temporary files. Files are  NOT backed up so this data should be moved to persistent storage once it is no longer needed for continued computation. Also note that these scratch directories may be cleaned nightly to purge files older than 30 days.

 

/scratch/lustre (shared, 27 TB)   is intended for dynamic data that requires higher bandwidth I/O. Files are NOTbacked up so this data should be moved to persistent storage as soon as it is no longer needed for computations.

 

/tigress-hsm (shared using GPFS, 270 TB)   is intended for more persistent storage and should provide high bandwidth i/o (400 MB/s aggregate bandwidth for jobs across 16 or more nodes). Users are provided with a default quota of 512 GB when they request a directory in this storage, and that default can be increased by requesting more. We do ask people to consider what they really need, and to make sure they regularly clean out data that is no longer needed since this filesystem is shared by the users of all our systems. See /tigress Usage Guidelines for more information.

 

/scratch (local to each compute node - 146 GB available on each node) is intended for data local to each task of a job, and it should be cleaned out at the end of each job.   This is the fastest storage for access.  Note that these scratch directories will be cleaned nightly to purge files older than 30 days.