Skip to content

Moving large files using Globus Connect to and from Sand

Jump to bottom
Zafer edited this page Oct 3, 2017 · 9 revisions

To move large files to and from Sand without using the WHSC TIC connection:

  • Create a Globus Online ID. Use your usgs.gov address for sign up.

  • Go to Globus Connect Personal, and login using your Globus Online ID (don't choose an organization).

  • Create an endpoint for your local Woods Hole machine, naming it something like #laptop (e.g. "rsignell#laptop"), and copy the endpoint key.

  • Install globus connect personal on your machine, and paste the endpoint key when prompted.

  • Try a transfer between your local machine and Sand, by going to https://www.globus.org/app/transfer, and entering in your local endpoint (e.g. "rsignell#laptop), and the Sand endpoint that has already been established as "usgscmg#sand". Authenticate the Sand endpoint using your username and password on Sand. If you don't know what it is, ask Rich.

  • Create an Globus Connect Personal endpoint on Yeti.
    Go to https://www.globus.org/app/endpoints/create-gcp, name it something like "rsignell#yeti", and copy the endpoint key.

For configuring Yeti, you have to connect to the TIC network, then do:

ssh yeti.cr.usgs.gov
wget https://s3.amazonaws.com/connect.globusonline.org/linux/stable/globusconnectpersonal-latest.tgz
tar xzf globusconnectpersonal-latest.tgz
cd globusconnectpersonal-x.y.z
./globusconnectpersonal -setup <endpoint-key> 
./globusconnectpersonal -start &

then modify the ~/.globusonline/lta/config-paths file to specify additional directories to make accessible. To share directories with read/write access, use the flags 0,1, so to add the /cxfs directory, it should look like this:

 more ~/.globusonline/lta/config-paths
~/,0,1
/cxfs,0,1

You should now be able to transfer files from local machine to Sand, and from Sand to Yeti. You cannot transfer files directly from local machine to Yeti, since the free globus account does not allow connections between two globus personal connect endpoints.

This is a good resource: https://www.osc.edu/resources/getting_started/howto/howto_transfer_files_using_globus_connect

To learn more about Globus, watch this short demo on Youtube.

Clone this wiki locally