Globus Online and BigJob

This guide aims to show how Globus Online can be used for file staging in BigJob and BigData. Please, review the Globus Online Documentation to understand how Globus Online works.

1. Setup Globus Connect Endpoint. The simplest way to do so is to utilize Globus Connect. The Mac OS X version provides a simple GUI that guides you through the process. In the following we describe how the Linux command line version is used.

   First an Globus Online Endpoint needs to be created using the following command:

    $ gsissh cli.globusonline.org 
   
    $ endpoint-add --gc egi
    Created the Globus Connect endpoint 'drelu#egi'.
    Use this setup key when installing Globus Connect: 
        9ee77dcd-282c-41c6-b6c5-80c4766fdad8
   

   Once the endpoint is created you can start Globus Connect by runnin:

     sh globusconnect -setup <setup key>
     sh globusconnect -start
   

2. The Globus Online Portal can be used to manage and activate Globus Online Endpoints. To utilize Globus Online e.g. in XSEDE context make sure you have setup your MyProxy service to utilize the XSEDE public endpoints.

3. BigJob relies on the Globus Online Python API. Currently, the package needs to be installed manually:

    easy_install globusonline-transfer-api-client
   

4. To utilize Globus Online, the working directory of BigJob needs to point to a Globus Online URL. The following conventions is used for this purpose go://<username>:<password>@globusonline.org?ep=<endpoint name>&path=<working directory>. For example:

    workingdirectory="go://<username>:<password>@globusonline.org?ep=xsede#lonestar4&path=~/bigjob/"
   

5. The same convention is used for the file transfer directives for bigjob and subjob:

    bj_filetransfers = ["go://<username>:<password>@globusonline.org?ep=drelu#MacBook&path=" 
                    + os.path.dirname(os.path.abspath(__file__)) 
                    + "/test.txt > BIGJOB_WORK_DIR"]
   
   
    jd.file_transfer = ["go://<username>:<password>@globusonline.org?ep=drelu#MacBook&path=" 
                   + os.path.dirname(os.path.abspath(__file__)) 
                   + "/test.txt > SUBJOB_WORK_DIR"]  
   

A full example can be found here.There are still some limitations, e.g. Globus Online does not allow transfers between two Globus Connect Endpoints.