Difference between revisions of "NAS"
| Line 55: | Line 55: | ||
jrwrigh7@lfe7: shiftc -r -d --sync /nobackup/jrwrigh7/models/STGFlatPlate/STFM_Tet_dz4-10_dx15 . | jrwrigh7@lfe7: shiftc -r -d --sync /nobackup/jrwrigh7/models/STGFlatPlate/STFM_Tet_dz4-10_dx15 . | ||
| − | This will copy the directory <code>STFM_Tet_dz4-10_dx15</code> to the current location. The flags do as follows | + | This will copy the directory <code>STFM_Tet_dz4-10_dx15</code> to the current location (<code>.</code>). The flags do as follows |
* <code>-r</code>: Recursively copy files from destination | * <code>-r</code>: Recursively copy files from destination | ||
| Line 63: | Line 63: | ||
Once this command is submitted, the transfer process will be backgrounded. Progress can be viewed by running <code>shiftc --monitor</code>. Additionally, you will recieve an email with the transfer job is completed. | Once this command is submitted, the transfer process will be backgrounded. Progress can be viewed by running <code>shiftc --monitor</code>. Additionally, you will recieve an email with the transfer job is completed. | ||
| + | |||
| + | It is also recommended to mirror the directory structure of your <code>/nobackup/$USER</code> directory on <code>lfe</code> to allow for the data to be easily recovered back to it's original state. This is especially important if you use symlinks (as they are path dependent). | ||
More documentation for <code>shiftc</code> can be found in its man page (<code>man shiftc</code>) and on [https://www.nas.nasa.gov/hecc/support/kb/shift-transfer-tool-overview_300.html NAS's documentation website]. | More documentation for <code>shiftc</code> can be found in its man page (<code>man shiftc</code>) and on [https://www.nas.nasa.gov/hecc/support/kb/shift-transfer-tool-overview_300.html NAS's documentation website]. | ||
Revision as of 09:11, 28 July 2020
Wiki for information related to the NASA Advanced Supercomputing (NAS) facility.
Contents
Overview
| Key | Value | Notes |
|---|---|---|
| Machines | Pleiades | Compute |
| Lou | Storage and Analysis | |
| Electra | Compute | |
| Endeavour | Compute | |
| Merope | Compute | |
| Job Submission System | PBS | |
| Facility Documentation | Support Knowledgebase |
How-To's
How-To's in Separate Wiki's
Backup Data from Scratch Directories
This is done simply by copying data from the /nobackup/$USER directories to your home directory on Lou (lfe). The /nobackup/$USER directories are mounted onto lfe, so transfers should be done on lfe.
This can be done with scp, but is recommended to use NASA's in-house utility shiftc. shiftc will automatically perform parallel file transfers, data integrity checks and repairs, and syncing features similar to rsync.
Commands:
jrwrigh7@lfe7: shiftc -r -d --sync /nobackup/jrwrigh7/models/STGFlatPlate/STFM_Tet_dz4-10_dx15 .
This will copy the directory STFM_Tet_dz4-10_dx15 to the current location (.). The flags do as follows
-
-r: Recursively copy files from destination -
-d: Create required directories that don't already exist. Equivalent of the-pflag formkdir -
--sync: Only copy over "new" files, where "new" are any changes to the modification time or file size.- If a file exists on destination (
.), but not source (STFM_Tet_dz4-10_dx15), it will not be copied back to source nor will it be deleted to match the state of source.
- If a file exists on destination (
Once this command is submitted, the transfer process will be backgrounded. Progress can be viewed by running shiftc --monitor. Additionally, you will recieve an email with the transfer job is completed.
It is also recommended to mirror the directory structure of your /nobackup/$USER directory on lfe to allow for the data to be easily recovered back to it's original state. This is especially important if you use symlinks (as they are path dependent).
More documentation for shiftc can be found in its man page (man shiftc) and on NAS's documentation website.
