{"id":137955,"date":"2021-12-03T15:27:12","date_gmt":"2021-12-03T20:27:12","guid":{"rendered":"http:\/\/www.bu.edu\/tech\/?page_id=137955"},"modified":"2023-08-25T12:48:20","modified_gmt":"2023-08-25T16:48:20","slug":"cloud-applications","status":"publish","type":"page","link":"https:\/\/www.bu.edu\/tech\/support\/research\/system-usage\/transferring-files\/cloud-applications\/","title":{"rendered":"Transferring Files using a Cloud Application"},"content":{"rendered":"

Overview<\/h2>\n

Data is stored in cloud storage services such as AWS S3 buckets, Google Drive, or One Drive. Alternatively, data may also be made available via Application Programming Interface (API) by the organization hosting the data. Accessing these sources require an internet connection and often an application to interact with the remote servers hosting the data.<\/p>\n

A dedicated node, called the Data Transfer Node, is available on the SCC for these types of data transfer tasks. The Data Transfer Node has its own high bandwidth internet connection and its only intended to be used for data transfer tasks. This page contains the following sections:<\/p>\n

    \n
  1. The Data Transfer Node<\/a><\/li>\n
  2. Submitting a Batch Job<\/a><\/li>\n
  3. Using SCC OnDemand for Interactive Data Transfer<\/a><\/li>\n
  4. Suggested Software and Modules for Transferring Data from Cloud Storage<\/a>\n
      \n
    1. lftp<\/a><\/li>\n
    2. Rclone<\/a><\/li>\n
    3. AWSCLI<\/a><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      <\/a>The Data Transfer Node<\/h2>\n

      The data transfer node is a single server named \u201cscc-globus.bu.edu<\/code>\u201d<\/span> that provides a direct 10GbE connection from the SCC to the internet for data transfer. The data transfer node is intended only for data transfer tasks and not for compute-intensive workloads. In comparison, the SCC compute nodes offer large computing capacity, but reside on a private network and share a connection to the outside world which limits the data transfer speeds. The data transfer node can be requested with the \u201c-l download<\/span><\/code>\u201d<\/span> command line option with a maximum runtime of 24 hours and limited to a single processor core.<\/p>\n

      <\/a>Submitting a Batch Job<\/h4>\n

      The -l download<\/span><\/code> command line option for qsub<\/span><\/code> will place your job on the data transfer node. This can be done for standard qsub<\/span><\/code> batch job submissions either on the command line or inside of a batch script<\/a>.<\/p>\n

      Example qsub<\/span><\/code> command line:<\/b><\/p>\n

      scc1$<\/span> qsub<\/span> \u2013P projectname<\/span> -l download<\/span> download.qsub<\/span><\/code><\/pre>\n
      Example batch script:<\/b>
      \nThis example script would use the data transfer node to synchronize files from Amazon Web Services S3 Service. A researcher should change the module and commands for their own purposes.<\/p>\n
      #!\/bin\/bash -l<\/span>\r\n\r\n#$ -P projectname<\/span>\r\n#$ -l download<\/span>\r\n\r\nmodule load<\/span> awscli<\/span>\r\naws<\/span> s3 sync s3:\/\/mybucket scc-directory<\/code><\/pre>\n
      <\/a>Using SCC OnDemand for Interactive Data Transfer<\/h2>\n
      Some data transfer programs must be run interactively, such as using the Firefox browser to access Google Drive cloud storage. For these applications, we recommend using a Desktop session on SCC OnDemand<\/a> for a remote desktop on the data transfer node. This can be done by selecting a \u201cDesktop\u201d interactive app, adding the \u201c-l download<\/span><\/code>\u201d<\/span> option to the \u201cExtra Qsub Options\u201d field and launching the job.<\/p>\n
      \"SCC<\/p>\n
      Once the job starts, you can connect to the desktop session and find yourself on the scc-globus.bu.edu<\/code><\/span> node with the ability to open web browsers or launch interactive download applications.<\/p>\n
      \"Black<\/p>\n
      <\/a>Suggested Software and Modules for Transferring Data from Cloud Storage<\/h2>\n
      There are many third party applications available that try to make transferring data from a cloud service provider easier by automating the task with only a few lines of commands. It is common for these tools to require upfront configuration to setup authentication in order to connect to a cloud service and access the data. Although we suggest these modules for your use, we do not guarantee they will work in all scenarios.<\/p>\n
      <\/a>lftp<\/h3>\n
      lftp<\/a> is a command line program that can transfer files from remote servers using FTP, SFTP, HTTP, and several other protocols. It is available as a system utility and does not require and modules to be loaded. lftp has a wide array of commands that can be used and its website has tutorials and documentation. Here is an example of using lftp to transfer an entire directory from a remote FTP server:<\/p>\n
      lftp -u username ftp.someserver.org\/path\/to\/remote\/directory<\/strong> -e \"cd path\/to\/scc\/destination<\/strong> ; mirror; exit\"<\/code><\/p>\n
      You will be prompted for the password on the remote server. This will copy the specified directory on the remote server to the destination you specify on the SCC.<\/p>\n
      <\/a>Rclone<\/h3>\n
      Rclone<\/a> is a command line program that allows one to configure a connection to over 40 cloud storage products. Use the following module load command to load rclone:<\/p>\n
      scc1$<\/span> module load rclone<\/span><\/code><\/pre>\n
      After loading the module, rclone needs to be configured to communicate with the cloud provider of your choice. Below are links to instructions on how to configure rclone with common cloud providers. Click on the link for the cloud provider you are connecting to and follow the instructions.<\/p>\n
      Note that the data transfer node does not accept interactive jobs from the scheduler; neither qsh<\/span><\/code> nor qrsh<\/span><\/code> requests will work with the -l download<\/span><\/code> command line option. For interactive use, researchers can SSH directly to scc-globus.bu.edu<\/code><\/span> from within the cluster or use SCC OnDemand<\/a>. The scc-globus node is dedicated to file transfers and interactive usage through SSH is subject to the same policies as apply on the login nodes.<\/p>\n