Janus Cheat Sheet

Login

 ssh your_identikey_user@login.rc.colorado.edu
 [pin][token]

Running Jobs

Environment Setup:

 use Torque
 use Moab
 use Git
 use .openmpi-1.4.3_intel-12.0_ib
 use Graphviz
 use Subversion

Viewing the list of dotkit modules (software) that you have loaded

 echo $_dk_inuse

available nodes/cores:

 pbsnodes -a
 showbf

Jobscript Template:

 #!/bin/bash
 #PBS -N name_of_job
 #PBS -l walltime=0:20:00
 #PBS -l nodes=1:ppn=12
 . /curc/tools/utils/dkinit
 reuse .openmpi-1.4.3_intel-12.0_ib
 
  mpirun /path/to/executable

Submission:

 qsub -q janus-debug jobscript.sh

Status

 checkjob job_id
 showq -u $USER

Cancel:

 qdel job_id

Large Memory, Interactive:

 qsub -I -l nodes=1:ppn=32,walltime=1:00:00 -q himem

Interactive, for debugging (with totalviewer)

 ssh -Y "user_id"@janus.rc.colorado.edu

 qsub -I -X -l nodes=1:ppn=12,walltime=1:00:00 -q janus-debug

Non Default allocation:

 qsub -A UCBXXXXXXX job.sh

or in the job script:

 #PBS -A UCBXXXXXXX

Debugging PHASTA

If you are getting hanging codes or crashing codes, compile with the pgi compilers and try the following flags on C, C++ and Fortran:

     -Ktrap=fp -Mbounds -Kieee
 

For PHASTA with cmake, you need to:

    CC=pgcc CXX=pgCC FC=pgfortran ccmake ../phasta
 

Then, add the compilation flags above to the following variables:

   CMAKE_CXX_FLAGS and CMAKE_C_FLAGS and CMAKE_Fortran_FLAGS variables
 

Basically this will make your code crash where you get floating point exceptions, go out of bounds with arrays, or have ieee exceptions. Then if you type

     set catchDebugger=1
 

and then

     mpirun  <exec>
 

it will launch debug windows which when you release them will stop at the point of your "foul" which REALLY helps.

 More info: http://www.pgroup.com/userforum/viewtopic.php?t=1898&postdays=0&postorder=asc&start=0&sid=719a9d9871bf16add8565a398ba548f3

Allocation Status

 module load utils/Crc-allocations
 check_allocation
 module load Moab
 showstats -u $USER

GridFTP

 export MYPROXY_SERVER_DN=$MYPROXY_SERVER_DN"/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443"
 myproxy-logon -T -b -s gridftp-00.rc.colorado.edu -v -l bema1643

 globus-url-copy -v -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -ss '/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443' gsiftp://gridftp-00.rc.colorado.edu/lustre/janus_scratch/bema1643/test.img sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/matthb2/foo/

globus-url-copy -v -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -ds '/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443' sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/mrasquin/416M/ gsiftp://gridftp-00.rc.colorado.edu/lustre/janus_scratch/bema1643/

(note: -ss to specify that the source server has a non-standard DN, -ds for the destination)

Queues/Policy

https://www.rc.colorado.edu/crcdocs/queues https://www.rc.colorado.edu/policies/janus-jobs