Computational Cluster Programs

Transferring files to/from an ATS-Hosted Cluster

UCLA/UC Grid Portals

You can use the UCLA Grid Portal to:

  • upload files from your local machine to the cluster.
  • download files and job output from the cluster to your local machine.
  • transfer files between the cluster and other clusters at UCLA.
The grid portal makes ATS-hosted clusters accessible from any web browser.

You can use the UC Grid Portal to:

  • transfer files between the cluster and clusters at other campuses of the University of California.

There is a file size limit of 1GB for files uploaded through the grid portals.

Globus Online

See High-speed file transfer with Globus Online for information on using Globus Online.

Globus Online is a software tool to transfer files across the web in a reliable, high-performance and secure way. It provides fault-tolerant, fire-and-forget data transfer using simple web or command line interfaces. It is approriate for transferring very large files either between your desktop machine and a remote machine like the Hoffman2 Cluster, or between two remote machines on which you have accounts.

Using scp or sftp

For security reasons, ATS-hosted clusters allow file transfer only with scp or sftp or grid-ftp. For the same reason, you should use an scp or sftp client on your local machine. You should not use the scp command on the cluster.

The scp and sftp commands transfer files using the secure shell protocol (ssh) in which data is encrypted during transfer. The use of scp requires that an scp client be run on the machine that you use to initiate the transfer and that it communicate with a server run on any other machines which participate in the transfer. ATS-hosted clusters, like most Linux and Unix systems, run both a client and a server.

There is an scp client command on desktop Linux/Unix systems and on Macs (use Terminal). On Windows, you usually have to install an ssh client which comes with an scp program. See Accessing an ATS-Hosted Cluster.

The syntax of the Linux/Unix scp command is very similar to the cp command. For complete scp syntax, enter:

man scp

Here is a symplified scp syntax that accomplishes most transfers:

scp [-r] source target

where source is the name of the file on your local machine, and target will be the name of the file on the cluster.

For the source on your local machine, specify an absolute or relative file name or directory name. You can use wild cards to transfer multiple files to an existing target directory. Specify -r to transfer a whole directory and its files.

For the target on the cluster, specify your login_id and its address, followed by a colon (:), followed by the file specification. You can specify the directory where the file is to be saved, or a dot "." meaning the same name in your home directory, or an absolute or relative path including a new file name.

login_id@hoffman2.idre.ucla.edu:filespec

For example:

scp myfile login_id@hoffman2.idre.ucla.edu:.

will transfer the file named myfile from your current directory on your local machine to your home directory on the Hoffman2 Cluster. Its name on the cluster will be $HOME/myfile

 

July 2011