Research Computing

The following instructions apply to della, hecate, mcmillan, orbital, orb, sesame1, sesame2, tiger1, tiger2, adroit, and nobel.
NOTE: DDT is not available on artemis or sesame3. Please let us know if it is needed on either of these.

• Start an instance of an X server on your personal system.
• ssh into the system's head node with X11-forwarding enabled. Depending on your settings, you may need to invoke the ssh client with "ssh -X".
• If not running on Nobel, then on the system's head node, in your .cshrc or .bashrc file:
  • Set up your module environment:
    • hecate: module load mpt/intel
    • mcmillan: module load cudatoolkit
    • other clusters: module load openmpi
  • Log out and log back in for it to take effect.
• Build your application as you normally would, but add the -g option to the mpicc, mpif90, icc, ifort, gcc (etc) command line so as to enable source-level debugging. Note that this will turn off compiler optimization, and your code will run slower.

• Type '/usr/licensed/bin/ddt'.
• The DDT window should now be displayed by your X server.
• If this is the first time you're running DDT on a given system, DDT will be configured with default values for that system. Change them as needed, as described below.
• In the 'DDT - Welcome' dialog box, click on 'Run and Debug a Program'.
• Select your Application, Arguments, Input File, and  Working Directory.
• If this is an MPI parallel application, then select the MPI checkbox.
  • Specify the number of nodes.
  • To verify the processes-per-node count, click on Change.
    • Select the Job Submission tab.
    • Adjust the PROCS_PER_NODE_TAG as needed.
    • Click OK.
• Set other  parameters as desired.
• On Nobel, click on Run; otherwise, click on Submit.

On Nobel, the job will start immediately.

On clusters other than Nobel, the debugging job will be submitted to the PBS batch queue, and the debugging session will automatically start when the job begins to run. Once the program has run to completion, you can restart the session without having to requeue the job in the PBS queue. But if you need to change any options or parameters, you will need to end the current session and start another, requiring a new job to be queued.

This approach is useful when the PBS batch job queue times are long, and allows a user to start one interactive PBS session, and then re-use it for multiple DDT sessions, as long as the number of resources needed do not exceed what was requested for the interactive PBS session.

• Type 'qsub -X -I -l nodes=X:ppn=Y,walltime=HH:MM:SS' (capital i, lower-case L).
• Wait for the interactive session prompt (on a compute node).
• Type '/usr/licensed/bin/ddt'.
• The DDT window should now be displayed by your X server.
• If this is the first time you're running DDT on a given system, DDT will be configured with default values for that system. Change them as needed.
• In the 'DDT - Welcome' dialog box, click on 'Run and Debug a Program'.
• Select your Application, Arguments, Input File, and  Working Directory.
• Click on Session | Options in the main DDT window.
• Click on Job Submission.
• Deselect "Submit job through queue".
• Click OK.
• If this is an MPI parallel application, then select the MPI checkbox.
  • Set "Number of processes" to X*Y.
• Set other  parameters as desired.
• Click Run.

Session will start immediately, and the processes will be distributed per the nodes and ppn specifications from the qsub command.

Allinea now has enhanced DDT to support debugging C or Fortran shared objects called from R (or presumably Python or other scripting languages). Sample instructions for debugging C code under R are as follows:

• Rebuild your C shared object (*.so file) with the compiler of your choice using its -g command line option. You may need to create/edit ~/.R/Makevars to do this. The '-g' option is used to support source-level debugging with DDT.
• Start up DDT as usual, and click on "Run and Debug a Program".
• For "Application", enter "/usr/lib64/R/bin/exec/R". This is the actual R executable that gets called by the /usr/bin/R script. DDT needs the real executable file, not a script.
• For "Arguments", enter "-f <your_R_script_path_and_name>".
• The various checkboxes should be clear, on the assumption that MPI is not being used.
• Click on "Environment Variables" and enter:
  R_HOME=/usr/lib64/R
  DDT_ENTRY_POINT=_start_ddt
• Click "Run", and ignore a couple of warning dialog boxes.
• In the main DDT window, right-click in the left-hand "Project Files" section, and select "Add file". Browse to the directory containing the C source file you want to debug, and click "Open".
• In the "project Files" section, click on the '+' next to "Source Files" to display your source file name.
• Double-click on your source file name to open it. Ignore any warning.
• Double-click on any line in your C source code to set a breakpoint. You will see a warning that the breakpoint will only be activated when your shared object is loaded. This is OK.
• Click on "Play", and the code will run until it reaches your breakpoint. You can now do DDT debugging as usual.

When DDT Memory Debugging is enabled, you may sometimes see the following warnings when debugging an OpenMPI application:

In order to avoid this warning and still allow OpenMPI to use the InfiniBand fabric, you will need to add the following parameters to the  mpirun arguments field of the DDT- Run dialog box:

--mca mpi_leave_pinned 0 --mca mpi_leave_pinned_pipeline 0