Konstantin's OpenPBS page

This page contains OpenPBS/Torque instructions for Fedora 3 and RHEL/SL 4.

Download

The sources, binaries and documentation are here.

Installation instructions

These are my notes from installing OpenPBS on the TWIST linux cluster, with PBS server on tw04.triumf.ca.

Node installation (updated 21-Apr-2006)

cd /triumfcs/mirror/fedora/linux/extras/3/i386/
rpm -vh --upgrade libtorque-2*.rpm torque-2*.rpm torque-mom-2*rpm torque-client-2*rpm torque-gui-2*rpm
killall pbs_mom
~olchansk/public_html/openpbs/pbsconfig tw04.triumf.ca

Server installation (not updated since 2003)

• install the openpbs binary rpm
• do the node installation
• configure pbs_server- run qmgr and issue these commands to create an execution queue and set server permissions:

  #
  # Create queues and set their attributes.
  #
  #
  # Create and define queue workq
  #
  create queue workq
  set queue workq queue_type = Execution
  set queue workq enabled = True
  set queue workq started = True
  #
  # Set server attributes.
  #
  set server acl_host_enable = True
  set server acl_hosts = *.triumf.ca
  set server managers = olchansk@*.triumf.ca
  set server managers += root@e614db.Triumf.CA
  set server default_queue = workq
  set server log_events = 127
  set server mail_from = adm
  set server query_other_jobs = True
  set server scheduler_iteration = 600
  Qmgr: 
  	

• create the nodes file, the nodes are allocated to jobs roughly in the same order as they are listed in this file:

  abc:ts np=2
  art:ts np=2
  karion:ts
  m13beam:ts np=2
  maher:ts
  midtwist:ts np=2
  buffalo:ts
  sam:ts np=2
  lin05:ts np=2
  lin06:ts np=2
  jam:ts np=2
  e614db:ts np=2
  	

• chkconfig pbs_server on
• /etc/rc.d/init.d/pbs_server restart

Scheduler installation

Configure either the stock pbs fifo scheduler per pbs instructions or the maui scheduler per maui instructions.

Maintenance tasks

Add a node

add node hostname to e614db:/etc/hosts.equiv qmgr create node nodename:ts np=numcpus
pbsnodes -c nodename
add "NODECFG[nodename] PARTITION=cpu1000" to /var/spool/maui/maui.cfg
service pbs_maui restart

Disable a node

pbsnodes -o nodename

Enable a node

pbsnodes -c nodename

Add node properties

qmgr set node nodename properties+=property

Run a job on a node with property "foo"

qsub -l nodes=1:foo ...

Clear "node busy"

qmgr set node nodename state=free

Contributed software

pbsconfig

Use pbsconfig for initial pbs node configuration. It records the server hostname, sets the pbs_mom permissions, probes the number of cpus and sets the node load thresholds.

Program usage:

Usage: pbsconfig [-h] [-s spooldir] servername
    

pbsrun

I found "qsub" and ilk ill suited for running existing programs and scripts "on the cluster", so I wrote pbsrun as a simple way to use pbs. With "pbsrun", the user command is executed in the current user directory, in the current user environment, as if there were no PBS and as if it were running on the local machine. The construction of PBS scripts, monitoring the progress of the job in the pbs queues, handling of pbs output files and cleanup is done behind the scenes.

For example, to run some user commands "on the cluster", one would do this:

Program usage:

Usage: pbsrun -N jobname "command... >& logfile" &
  Options: 
    [-h] Prints out this message.
    [-N job_name]     name this job.
    [-m user@host]   emails you when the job begin, ends or aborts.
    [-o qsub_options] passes options to the call to qsub.
    [-q queue]        selects a queue.
    [-n] test mode: do not run any pbs commands.
    [-k] keep, do not delete temp files.
    [-v] be verbose, show progress.