How to Transfer Large Files Over SSH in Ubuntu

Sometimes, sharing files between computers feels like a breeze. But when it comes to large files, the task can seem like a mountain climb. But don’t worry! In this guide, I’ll walk you through transferring large files over SSH in Ubuntu – one of the easiest, most secure methods out there. Plus, if you’re working with big files for AI tools or system backups, you’ll see how helpful this can be.

Why Transfer Large Files Over SSH?

SSH, or Secure Shell, is like a digital handshake – it lets two computers communicate in a safe, encrypted way. Think of it like a sealed envelope that only the recipient can open. The process is simple, and because it’s secure, you don’t need to worry about someone snooping on your data.

For Linux and Ubuntu users, SSH is especially useful. You get to keep your data secure, and it’s super efficient – a win-win! So whether you’re transferring AI models, Linux distribution images, or simply large backup files, SSH has your back.

Prerequisites: What You Need to Get Started

Before we get into the methods, here’s a quick list of what you need:

1. SSH Server Access: Make sure SSH is enabled on both computers.
2. Terminal Access: You’ll be using the command line for most methods.
3. Basic Commands: Some familiarity with commands like scp, rsync, and tar will be helpful.

Alright, let’s dive into some methods that make file transfer over SSH in Ubuntu easy as pie!

Method 1: Using SCP (Secure Copy Protocol)

SCP, or Secure Copy Protocol, is one of the most straightforward ways to transfer files over SSH. It’s fast, secure, and already built into Ubuntu, so no extra installations are needed.

Step 1: Basic SCP Command

Here’s the basic format for using SCP:

scp [options] /path/to/local/file username@remote_host:/path/to/remote/directory

• /path/to/local/file is where your file is stored.
• username@remote_host is the login details of the destination system.
• /path/to/remote/directory is where you want the file to go.

Example:

Say you have a large file called dataset.zip on your local machine, and you want to transfer it to the remote server:

scp dataset.zip your_username@remote_server:/home/your_username/

It’s really that simple! SCP encrypts the file during the transfer, so you don’t have to worry about security.

Step 2: Transfer an Entire Folder

Need to transfer a whole folder? Just add the -r option:

scp -r /path/to/local/folder username@remote_host:/path/to/remote/directory

This command copies everything inside the folder, so you don’t need to worry about each individual file.

Tips for Using SCP

• Use -P to specify the port if your SSH server is running on a different port. For example, scp -P 2222.
• Consider compressing files before transfer to speed up the process (more on that in a minute!).

Method 2: Using Rsync for Efficient Transfers

If you’re looking to transfer large files or entire directories, Rsync is a fantastic option. It’s like SCP but with some extra smarts – Rsync only transfers parts of files that have changed, saving bandwidth and time.

Step 1: Basic Rsync Command

Here’s the basic Rsync format:

rsync -avz /path/to/local/file username@remote_host:/path/to/remote/directory

• -a preserves the file permissions.
• -v makes it verbose (it’ll show you what’s happening).
• -z compresses the data to make the transfer faster.

Example:

Let’s say you’re transferring a large project folder called my_project:

rsync -avz my_project/ your_username@remote_server:/home/your_username/

This command will zip through the files, only sending data where it’s needed, making it perfect for large, complex folders.

Step 2: Resume Interrupted Transfers

If your internet connection drops, don’t worry! Rsync can resume transfers with the --partial flag:

rsync -avz --partial my_project/ your_username@remote_server:/home/your_username/

This option is a lifesaver if you’re transferring big files over a shaky connection.

Method 3: Compress Files for Faster Transfer

When dealing with huge files (especially with data-heavy tasks like AI or machine learning), it helps to compress files before transferring them. This method can shave off a ton of time.

Step 1: Compress Your File with tar

The tar command combines files into one compressed file. Here’s how you do it:

tar -czvf archive_name.tar.gz /path/to/directory_or_file

• -c creates the archive.
• -z compresses it using gzip.
• -v gives you a verbose output (handy for tracking progress).
• -f specifies the file name.

Example:

To compress a folder named big_folder, you’d do:

tar -czvf big_folder.tar.gz big_folder/

Step 2: Transfer the Compressed File

Once your file is compressed, transfer it using SCP or Rsync as we covered. On the receiving end, you can extract it with:

tar -xzvf big_folder.tar.gz

Advanced Tip: Use Mosh for Long-Distance or Unstable Connections

If you’re dealing with long-distance connections or a shaky network, Mosh (Mobile Shell) can be a game-changer. Mosh works with SSH but is much better at handling spotty connections and network changes.

Here’s the basic command to start a Mosh session:

mosh username@remote_host

Once connected, you can use Rsync or SCP like usual, with the added benefit of Mosh’s stability.

Note: You may need to install Mosh on both the local and remote machines first.

Practical Tips for Faster Transfers

1. Use Compression: Always compress files if they’re large. It can save you a surprising amount of time.
2. Limit Bandwidth: If you don’t want to hog all your internet, add --bwlimit to Rsync to cap the bandwidth.
3. Run in Background: If you need to transfer huge files, try running the command in the background by appending & at the end, so you can still use the terminal.

Frequently Asked Questions

1. Can I use SSH to transfer files between two remote servers?

Yes! Just connect via SSH to one server, then use SCP or Rsync to copy files to the second server. It’s like a chain link – but make sure you have access to both servers.

2. Which is faster: SCP or Rsync?

In most cases, Rsync is faster because it only transfers changed parts of files. However, for a one-time transfer, SCP can be quicker since it’s simpler.

3. How can I check transfer progress?

Both Rsync and SCP have options for this. For Rsync, use the -P flag, and for SCP, you can check progress by using pv (a separate tool) if you want detailed output.

Wrapping Up

Transferring large files over SSH in Ubuntu might sound intimidating, but once you get the hang of it, it’s a piece of cake. Whether you’re moving files between your local and remote machines or juggling big datasets for machine learning, using SSH ensures everything is safe, fast, and efficient.

Now it’s your turn! Give these methods a try, and soon enough, transferring files over SSH will feel like second nature.