GridFTP

Introduction

GridFTP is a high performance file transfer tool supported by many of the large super-compute sites. It uses parallelism to hide the effects of latency and TCP's congestion controls.

GridFTP is part of the larger Globus Toolkit, which also includes (among other things) a custom PKI based authentication mechanism. This can be used to achieve password-less authentication with supported sites (most of TeraGrid for example).

Setup

GridFTP should already be in your path and configured by default on jumpgate-phasta.colorado.edu. If it is not (due to unusual shell configuration or other issues) you can usually get it by running

 source /etc/profile

To test if you have GridFTP in your path correctly you can run these command (each should give you some output)

 which globus-url-copy
 which myproxy-logon
 echo $GLOBUS_LOCATION
 echo $LD_LIBRARY_PATH | grep gt5

To use sshftp mode, you'll also want to have a SSH preshared key configured with your account, (either without a password, or with ssh-agent)

Authentication

Most sites support the use of "GSI" password-less authentication. If the site you want to connect to offers this option, you probably want to use it.

The way GSI authentication is usually implemented uses a tool called "myproxy" which generates a key which you can use for a short time to authenticate without a password.

The first step to using GSI authentication is to install the necessary certificates in your home directory. Most sites will allow you to do this automatically using the myproxy tool's "-T" option. You only need to use this option the first time you use myproxy.

For example, to configure my account to connect to the GridFTP/myproxy server at ANL, I could use this command:

 myproxy-logon -T -l matthb -t 300 -s gs1.intrepid.alcf.anl.gov

This command can be disected as follows: -T tells myproxy to fetch the trust roots (CA, etc). You should only need to do this the first time you connect to a particular server and when the server's CA is updated.

-l matthb specifies my username (replace "matthb" with your username)

-t 300 specifies that I want my session key to expire in 300 hours (the default is 24)

-s gs1.intrepid.alcf.anl.gov specifies the myproxy server (*not* necessarily the gridftp server)

Common myproxy Servers

Intrepid and Eureka (ALCF/ANL)

 gs1.intrepid.alcf.anl.gov

(authenticate with your CryptoCard and pin)

Teragrid (Kraken, Ranger, etc)

 myproxy.teragrid.org

(authenticate with your Teragrid portal username/password)

• -phasta.colorado.edu

 myproxy-phasta.colorado.edu

(authenticate with your normal UNIX/LDAP username/password) (coming soon)

GSISSH

Once you have a GSI key/ticket, many sites allow you to use it to gain shell access as well as do file transfers. For example, even if I don't have my SecureID for Kraken handy, I can still log-in using GSI and my TeraGrid credentials as follows:

 ssh jumpgate-phasta.colorado.edu
 myproxy-logon -T -l myteragriduser -s myproxy.teragrid.org
 gsissh mykrakenuser@kraken-gsi.nics.utk.edu

GridFTP

The primary tool for doing GridFTP transfers is called "globus-url-copy" To see it's complete usage you can run

 globus-url-copy -help

In general, you should start with the following set of options:

 globus-url-copy -r -cd -rst -rst-retries 0 -fast -vb -p 64 -g2 -stripe -tcp-bs 4M

followed by your source and destination URLs.

To transfer to/from jumpgate-phasta.colorado.edu your URL should look something like this

sshftp://username@jumpgate-phasta.colorado.edu/users/username/file

Please see the site specific documentation for other site's URLs. These URLs may be helpful to get you started: https://wiki.alcf.anl.gov/index.php/Using_GridFTP https://www.teragrid.org/web/user-support/transfer_location http://www.olcf.ornl.gov/kb_articles/gridftp/

Recent versions of globus-url-copy also support a mode which behaves somewhat like rsync, enabled by the

 -sync -sync-level 3

flags.

A complete example to copy a directory from ALCF to jumpgate would be:

 globus-url-copy -r -cd -rst -rst-retries 0 -fast -vb -p 64 -stripe -tcp-bs 4M -g2 -sync -sync-level 3 gsiftp://gridftp.intrepid.alcf.anl.gov/intrepid-fs0/users/matthb/scratch/ sshftp://matthb2@jumpgate-phasta.colorado.edu/scratch/matthb2/scratch_from_anl/

UberFTP

A simpler (but less powerful) tool to do GridFTP transfers is UberFTP. First get a certificate from myproxy as before then run:

 uberftp hostname

Set the number of streams with

 parallel 128

then interact as you would with traditional ftp. The full list of commands is available by typing "help"

 UberFTP> help
 Usage "help [topic]" where topic is one of:
 !               ?               active          ascii             binary          
 blksize         bugs            bye             cat             cd              
 cdup            chgrp           chmod           cksum             close           
 dcau            debug           dir             family            get             
 glob            hash            help            keepalive          lcat            
 lcd             lcdup           lchgrp          lchmod            lclose          
 ldir            lls             lmkdir          lopen             lpwd            
 lquote          lrename         lrm             lrmdir          ls              
 lsize           lstage          mget            mkdir           mode            
 mput            open            order           parallel        passive         
 pbsz            pget            pput            prot            put             
 pwd             quit            quote           rename          resume          
 retry           rm              rmdir           runique           size            
 stage           sunique         tcpbuf          versions            wait            

You probably will need at least the "ls", "get", "put", and "mget"/"mput" commands

GridFTP Tuning

Depending on the network conditions, you can see massive GridFTP can perform very well or very poorly. You can often improve things by changing a few parameters:

 -p 64

Specifies that you want to use 64 streams. Because TCP needs to receive an acknowledgment before more data can be sent latency can drastically reduce performance over long distances. By adding more streams you can increase the amount of in-transit data and somewhat hide the effects of latency. If performance is poor try increasing this number until you stop seeing improvement. Using excessive numbers of streams won't help and will increase the load on the servers involved. You shouldn't need more than one or two more doublings of the value recommended here.

 -pp

Enable pipelining. This can help with large number of files (particularly relatively small ones), but can also be buggy. If your transfers hang, try disabling this. If you're transferring lots of files, try enabling it.

 -cc 4

Split the transfer over several (4 in this case) concurrent GridFTP sessions. This can also help with large numbers of small files. Use sparingly, and disable if you experience stalled/crashed transfers.

 -sync-level 3

If you're using "sync" mode (similar to rsync) this determines the algorithm used to decide which files need to be (re) transferred. Level 3 uses checksums and will be the most reliable, but slowest. Level 2 is probably fine for most uses. See the globus-url-copy documentation for all the options.

More Examples

Kraken:

 myproxy-logon -T -l teragriduser -s myproxy.teragrid.org
 globus-url-copy -dbg -r -cd -rst -rst-retries 0 -pp -fast -vb -p 64 -stripe -tcp-bs 4M -g2 gsiftp://user@gridftp.kraken.nics.teragrid.org:2811/lustre/scratch/bmatth/test.img sshftp://user@jumpgate-phasta.colorado.edu/scratch/user/

Janus

Script to transfer data (tar or directory) between jumpgate-phasta and Kraken/Intrepid/Janus (easy to update for other destinations):

  transfer.sh

 example: ./transfer.sh 1 kraken2ucb /lustre/scratch/mrasquin/Models/Boeing/Beta/Open6_B5_D20_U20/15-A0-12jets/Runs-12jets/Archive4992.3600-3780/ /users/mrasquin/Models/Boeing/Beta/Open6_B5_D20_U20/15-A0-12jets/Runs-12jets/Archive4992.3600-3780/ 1

You have to connect to jumpgate-phasta to execute this script. Do not forget to replace "mrasquin" by your TeraGrid login name (for the "myproxy-logon" command) and jumpgate-phasta and kraken login name (for the "globus-url-copy" commands).

Note that it is faster to transfer a tar file than a directory that contains many files. But to tar or untar an archive on Kraken or Intrepid can take much more time so that it is recommended (so far) not to tar any directory and transfer its content directly.

[| SCOREC]

Troubleshooting

If you get a message from myproxy that looks like this, try adding the "-b" option as prompted the (you should only need this option the first time you connect)

 Error authenticating: GSS Major Status: Authentication Failed
 GSS Minor Status Error Chain:
 globus_gss_assist: Error during context initialization
 OpenSSL Error: s3_clnt.c:985: in library: SSL routines, function SSL3_GET_SERVER_CERTIFICATE:   certificate verify failed  
 globus_gsi_callback_module: Could not verify credential
 globus_gsi_callback_module: Can't get the local trusted CA certificate: Untrusted self-signed   certificate in chain with hash 9b95bbf2
 
 The CA that signed the myproxy-server's certificate is untrusted.
 If you want to trust the CA, re-run with the -b option.

You command would then look something like this

 myproxy-logon -s myproxy.teragrid.org -l matthb2 -T -b