Janus Cheat Sheet

Login

 ssh your_identikey_user@login.rc.colorado.edu
 [pin][token]

Running Jobs

Environment Setup:

 use Torque
 use Moab
 use Git
 use .openmpi-1.4.3_intel-12.0_ib
 use Graphviz
 use Subversion

available nodes/cores:

 pbsnodes -a
 showbf

Jobscript Template:

 #!/bin/bash
 #PBS -N name_of_job
 #PBS -l walltime=0:20:00
 #PBS -l nodes=1:ppn=12
 . /curc/tools/utils/dkinit
 reuse .openmpi-1.4.3_intel-12.0_ib
 
  mpirun /path/to/executable

Submission:

 qsub -q janus-debug jobscript.sh

Status

 checkjob job_id
 showq -u $USER

Cancel:

 qdel job_id

Large Memory, Interactive:

 qsub -I -l nodes=1:ppn=32,walltime=1:00:00 -q himem

Interactive, for debugging (with totalviewer)

 qsub -X -I -l nodes=1:ppn=12,walltime=1:00:00 -q janus-debug

Non Default allocation:

 qsub -A UCBXXXXXXX job.sh

or in the job script:

 #PBS -A UCBXXXXXXX

Debugging PHASTA

 If you are getting hanging codes or crashing codes, compile with the pgi compilers and try the following flags on  C, C++ and Fortran:
 
 -Ktrap=fp -Mbounds -Kieee
 
 Basically this will make your code crash where you get floating point exceptions, go out of bounds with arrays, or have ieee exceptions. Then if you type
 
 set catchDebugger=1
 
 and then 
 
 mpirun  <exec>
 
 it will launch debug windows which when you release them will stop at the point of your "foul" which REALLY helps.

Allocation Status

 use Crc-allocations
 check_allocation.py
 use Moab
 showstats -u $USER

GridFTP

 export MYPROXY_SERVER_DN=$MYPROXY_SERVER_DN"/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443"
 myproxy-logon -T -b -s gridftp-00.rc.colorado.edu -v -l bema1643

 globus-url-copy -v -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -ss '/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443' gsiftp://gridftp-00.rc.colorado.edu/lustre/janus_scratch/bema1643/test.img sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/matthb2/foo/

globus-url-copy -v -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -ds '/C=US/O=Globus Consortium/OU=Globus Connect Service/CN=842a610a-4de7-11e1-9674-123138151443' sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/mrasquin/416M/ gsiftp://gridftp-00.rc.colorado.edu/lustre/janus_scratch/bema1643/

(note: -ss to specify that the source server has a non-standard DN, -ds for the destination)

Queues/Policy

https://www.rc.colorado.edu/crcdocs/queues https://www.rc.colorado.edu/policies/janus-jobs