GridFTP
Contents
Introduction
GridFTP is a high performance file transfer tool supported by many of the large super-compute sites. It uses parallelism to hide the effects of latency and TCP's congestion controls.
GridFTP is part of the larger Globus Toolkit, which also includes (among other things) a custom PKI based authentication mechanism. This can be used to achieve password-less authentication with supported sites (most of TeraGrid for example).
Setup
GridFTP should already be in your path and configured by default on jumpgate-phasta.colorado.edu. If it is not (due to unusual shell configuration or other issues) you can usually get it by running
source /etc/profile
To test if you have GridFTP in your path correctly you can run these command (each should give you some output)
which globus-url-copy which myproxy-logon echo $GLOBUS_LOCATION echo $LD_LIBRARY_PATH | grep gt5
To use sshftp mode, you'll also want to have a SSH preshared key configured with your account, (either without a password, or with ssh-agent)
Authentication
Most sites support the use of "GSI" password-less authentication. If the site you want to connect to offers this option, you probably want to use it.
The way GSI authentication is usually implemented uses a tool called "myproxy" which generates a key which you can use for a short time to authenticate without a password.
The first step to using GSI authentication is to install the necessary certificates in your home directory. Most sites will allow you to do this automatically using the myproxy tool's "-T" option. You only need to use this option the first time you use myproxy.
For example, to configure my account to connect to the GridFTP/myproxy server at ANL, I could use this command:
myproxy-logon -T -l matthb -t 300 -s gs1.intrepid.alcf.anl.gov
This command can be disected as follows: -T tells myproxy to fetch the trust roots (CA, etc). You should only need to do this the first time you connect to a particular server and when the server's CA is updated.
-l matthb specifies my username (replace "matthb" with your username)
-t 300 specifies that I want my session key to expire in 300 hours (the default is 24)
-s gs1.intrepid.alcf.anl.gov specifies the myproxy server (*not* necessarily the gridftp server)
Common myproxy Servers
Intrepid and Eureka (ALCF/ANL)
gs1.intrepid.alcf.anl.gov
(authenticate with your CryptoCard and pin)
Teragrid (Kraken, Ranger, etc)
myproxy.teragrid.org
(authenticate with your Teragrid portal username/password)
- -phasta.colorado.edu
myproxy-phasta.colorado.edu
(authenticate with your normal UNIX/LDAP username/password) (coming soon)
GSISSH
Once you have a GSI key/ticket, many sites allow you to use it to gain shell access as well as do file transfers. For example, even if I don't have my SecureID for Kraken handy, I can still log-in using GSI and my TeraGrid credentials as follows:
ssh jumpgate-phasta.colorado.edu myproxy-logon -T -l myteragriduser -s myproxy.teragrid.org gsissh mykrakenuser@kraken-gsi.nics.utk.edu
GridFTP
The primary tool for doing GridFTP transfers is called "globus-url-copy" To see it's complete usage you can run
globus-url-copy -help
In general, you should start with the following set of options:
globus-url-copy -r -cd -rst -rst-retries 0 -fast -vb -p 64 -g2 -stripe -tcp-bs 4M
followed by your source and destination URLs.
To transfer to/from jumpgate-phasta.colorado.edu your URL should look something like this
sshftp://username@jumpgate-phasta.colorado.edu/users/username/file
Please see the site specific documentation for other site's URLs. These URLs may be helpful to get you started: https://wiki.alcf.anl.gov/index.php/Using_GridFTP https://www.teragrid.org/web/user-support/transfer_location http://www.olcf.ornl.gov/kb_articles/gridftp/
Recent versions of globus-url-copy also support a mode which behaves somewhat like rsync, enabled by the
-sync -sync-level 3
flags.
A complete example to copy a directory from ALCF to jumpgate would be:
globus-url-copy -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -sync -sync-level 3 gsiftp://gridftp.intrepid.alcf.anl.gov/intrepid-fs0/users/matthb/scratch/ sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/matthb2/scratch_from_anl/
UberFTP
A simpler (but less powerful) tool to do GridFTP transfers is UberFTP. First get a certificate from myproxy as before then run:
uberftp hostname
Set the number of streams with
parallel 128
then interact as you would with traditional ftp. The full list of commands is available by typing "help"
UberFTP> help Usage "help [topic]" where topic is one of: ! ? active ascii binary blksize bugs bye cat cd cdup chgrp chmod cksum close dcau debug dir family get glob hash help keepalive lcat lcd lcdup lchgrp lchmod lclose ldir lls lmkdir lopen lpwd lquote lrename lrm lrmdir ls lsize lstage mget mkdir mode mput open order parallel passive pbsz pget pput prot put pwd quit quote rename resume retry rm rmdir runique size stage sunique tcpbuf versions wait
You probably will need at least the "ls", "get", "put", and "mget"/"mput" commands
GridFTP Tuning
Depending on the network conditions, you can see massive GridFTP can perform very well or very poorly. You can often improve things by changing a few parameters:
-p 64
Specifies that you want to use 64 streams. Because TCP needs to receive an acknowledgment before more data can be sent latency can drastically reduce performance over long distances. By adding more streams you can increase the amount of in-transit data and somewhat hide the effects of latency. If performance is poor try increasing this number until you stop seeing improvement. Using excessive numbers of streams won't help and will increase the load on the servers involved. You shouldn't need more than one or two more doublings of the value recommended here.
-pp
Enable pipelining. This can help with large number of files (particularly relatively small ones), but can also be buggy. If your transfers hang, try disabling this. If you're transferring lots of files, try enabling it.
-cc 4
Split the transfer over several (4 in this case) concurrent GridFTP sessions. This can also help with large numbers of small files. Use sparingly, and disable if you experience stalled/crashed transfers.
-sync-level 3
If you're using "sync" mode (similar to rsync) this determines the algorithm used to decide which files need to be (re) transferred. Level 3 uses checksums and will be the most reliable, but slowest. Level 2 is probably fine for most uses. See the globus-url-copy documentation for all the options.
More Examples
Kraken:
myproxy-logon -T -l teragriduser -s myproxy.teragrid.org globus-url-copy -dbg -r -cd -rst -rst-retries 0 -pp -fast -vb -p 64 -stripe -tcp-bs 4M -g2 gsiftp://user@gridftp.kraken.nics.teragrid.org:2811/lustre/scratch/bmatth/test.img sshftp://user@jumpgate-phasta.colorado.edu/scratch/user/
Script to transfer data (tar or directory) between jumpgate-phasta and Kraken/Intrepid/Janus (easy to update for other destinations):
transfer.sh
example: ./transfer.sh 1 kraken2ucb /lustre/scratch/mrasquin/Models/Boeing/Beta/Open6_B5_D20_U20/15-A0-12jets/Runs-12jets/Archive4992.3600-3780/ /users/mrasquin/Models/Boeing/Beta/Open6_B5_D20_U20/15-A0-12jets/Runs-12jets/Archive4992.3600-3780/ 1
You have to connect to jumpgate-phasta to execute this script. Do not forget to replace "mrasquin" by your TeraGrid login name (for the "myproxy-logon" command) and jumpgate-phasta and kraken login name (for the "globus-url-copy" commands).
Note that it is faster to transfer a tar file than a directory that contains many files. But to tar or untar an archive on Kraken or Intrepid can take much more time so that it is recommended (so far) not to tar any directory and transfer its content directly.
[| SCOREC]
Troubleshooting
If you get a message from myproxy that looks like this, try adding the "-b" option as prompted the (you should only need this option the first time you connect)
Error authenticating: GSS Major Status: Authentication Failed GSS Minor Status Error Chain: globus_gss_assist: Error during context initialization OpenSSL Error: s3_clnt.c:985: in library: SSL routines, function SSL3_GET_SERVER_CERTIFICATE: certificate verify failed globus_gsi_callback_module: Could not verify credential globus_gsi_callback_module: Can't get the local trusted CA certificate: Untrusted self-signed certificate in chain with hash 9b95bbf2 The CA that signed the myproxy-server's certificate is untrusted. If you want to trust the CA, re-run with the -b option.
You command would then look something like this
myproxy-logon -s myproxy.teragrid.org -l matthb2 -T -b
