View source for ALCF/Archiving Data at ALCF

[[ALCF]]'s High Performance Storage System (HPSS) is a robotic tape drive system used for large amount of archival data storage that will not be often accessed. The system has two interfaces listed in ALCF documentation ([https://www.alcf.anl.gov/support-center/theta/using-hpss-theta]) that can be used, <code>hsi</code> and <code>htar</code>. In this wiki, the <code>hsi</code> interface will be focused on. 

== HSI Basics ==

<code>hsi</code> is a utility to interface with the HPSS system. It looks and operates much like the typical bash command lines that we are used to, but with some added complexities. When you enter <code>hsi</code>, the system will place you into your home HPSS space at <code>/home/username</code>. <code>hsi</code> keeps track of both your location in this HPSS space and also your location in the "local" system that you are running <code>hsi</code> from. The <code>hsi</code> system will automatically set the "local" directory location to be the location that you entered the utility from. 

Navigation through HPSS operates the same as a normal command line, with <code>ls</code> <code>cd</code>, and <code>mkdir</code> among others being valid commands for navigation through HPSS. If you need to navigate though the "local" directories though, this can still be done by appending an "l" to the front of these standard commands (i.e. <code>lls</code> <code>lcd</code>, and <code>lmkdir</code>). This can be useful if you entered <code>hsi</code> at the incorrect point or wish to archive data in multiple locations.

It should be noted that <code>hsi</code> does support tab-complete or up-arrowing for past commands. It is recommended that you enter the utility with a defined plan in order to reduce the amount of annoyance that the lack of these luxuries can cause.

While standard use cases of <code>hsi</code> will be covered below, more documentation (that is more thorough and helpful than the ALCF documentation is available here: [https://docs.nersc.gov/filesystems/archive/]. 

== Archiving of Data ==

Once the destination directory for the data has been created and/or navigated to, there are a few options and considerations to actually archive the data. These are most notably:

* <code>put</code>
* <code>cput</code>

<code>put</code> is the most basic archiving tool, and will overwrite any versions of the files being archived already on HPSS. <code>cput</code> is a conditional version of <code>put</code> that will only overwrite files if there is a newer version "locally" compared to the file already on HPSS. This makes <code>cput</code> the tool of choice for updating partially archived datasets, but due to its otherwise similar functionality to <code>put</code>, it is also the recommended default command to use.

Both <code>put</code> and <code>cput</code> have similar syntax, and the following will cover both, but <code>cput</code> will be used as an example. It is assumed that the <code>hsi</code> command has already been run to enter the <code>hsi</code> utility before attempting the following.

Simple usage to store a single file is:

<code>cput <filename></code>

To change the name of a file as it is archived:

<code>cput <filename> : <newFilename></code>

Whole directories can be stored using:

<code>cput -R <dirName></code>

If you with to keep the parent directory intact, or with:

<code>cput -R "*"</code>

If you simply want to move the contents of a directory and everything beneath it. Simply be mindful of your "local" directory location when choosing between these options.

If you need to retrieve data from tape and put it back on the "local" system, the <code>get</code> and <code>cget</code> commands act in the same way as <code>put</code> and <code>cput</code> but in reverse.

== Troubleshooting ==
If you are a first time user of HPSS, you will likely get an error regarding a key file. This is something that must be taken care of by ALCF support (support@alcf.anl.gov). Simply email them with your ALCF username and state that you need access set up for HPSS.

[[Category:Compute Facilities]]