PBSPro Queue System
Cortex utilizes the PBSPro queue system from Altair Engineering. The complete user's guide for PBSPro version 5.4.2 is available here. In addition to the full User's Guide and the howto posted here, the PBSPro Quick Start Guide is also available here. If you have any questions or problems with the queue system on cortex, please email lanceh@cse.unr.edu.
Writing a PBS Job Script
After compiling your MPI program as described in the MPI Help Documentation guide, it is necessary to write a simple script containing the mpirun command and a few other bits of information so that the queue system can make descisions about how to schedule and run your job. Below is a simple example and explanation.
#!/bin/bash
#PBS -l nodes=4:cpp=2
#PBS -m be
/opt/mpich/gnu/bin/mpirun -np 8 -machinefile $PBS_NODEFILE a.out
The above script shows how to use #PBS directives to educate the queue system about your job. Note that lines beginning with #PBS will be treated as comments by the shell, in this case BASH, but not by the PBS Queue system! The PBS -l nodes=4:cpp=2 line in this script specifies to PBS that this job needs 4 nodes, 2 processors per node, for a total of 8 processors. All nodes in the cortex cluster are dual-processor nodes. The #PBS -m be option tells PBS to email me when the job begins and finishes executing. All mail is currently sent to the user's local mailbox. After giving this basic information to PBS, we add our job executable and any other information to the script. Please note that it is necessary to pass the environment variable $PBS_NODEFILE to the mpirun command above, as this is how the queue can dynamically assign nodes to your job. Do not incude a path to a machine file in your directory or the queue will not run your job. In the example case, we execute the MPI program a.out using mpirun. This basic information should be sufficient for most all job submissions made to PBS. Save the script you have made and then it can be submitted to the queue for running.
Submitting Your Job
To submit your job to the queue, type qsub path to job script. For example, qsub runme.mpi.
Checking Your Job's Status
There are numerous ways to check on your jobs status after submitting it to the queue. The easiest is with the qstat command. This text based status tool will show the status of all jobs in the queue. There are also two graphical tools located in /usr/pbs/bin named xpbsmon and xpbs that can be used to show the status of the queues on cortex. More information on these tools is available in the full user's guide located at the link above.
Getting Your Results
The standard output and standard error from your job will be placed into your home directory as jobname.o and jobname.e files, respectively. A number may be appended after the .o and .e, this number is your job's PBS job number assigned upon job submission.