Automating Backups for Your AI Projects on Linux

Step‑by‑step guide (2025)

Why Backup Matters

If you’re a data scientist, hobbyist, or student who’s just started tinkering with machine learning, the biggest headache isn’t the model architecture. It’s the fact that your workstation is a single point of failure. One power outage, one accidental “rm -rf /”, and all your notebooks, datasets, and trained weights are gone.

I built my own AI‑ready Linux workstation for under $800 last year: an AMD Ryzen 5 5600X, an NVIDIA RTX 3060, a fast NVMe SSD, and a secondary SATA drive for backups. After setting up CUDA, LM Studio, and Golem for distributed training, I hit “Run” and realized the real test began: How do I keep all that data safe?

This guide walks you through automating backups on Linux—hardware choices, OS setup, GPU drivers, and a fully‑automated snapshot pipeline—so you can focus on building models instead of fearing data loss.

1. Pick Reliable Hardware (and Save)

| Component | Why It Matters | Affiliate | — |-----------|----------------|-----------| — | Motherboard with SATA & NVMe | Allows adding a secondary drive later without buying another board. | — |
| Dual‑BIOS / BIOS recovery | Prevents boot issues if you accidentally flash the wrong firmware. | — | — | 80+ Bronze PSU | Reduces risk of sudden power loss that could corrupt drives. | — | — | NVMe SSD (500 GB+) for OS & training data | Faster I/O speeds mean less time waiting during backup. | — |
| Secondary SATA SSD or HDD (1‑2 TB) for backups | Keeps your primary drive free from wear and acts as a quick restore point. | — |

Tip: Even on a tight budget, invest in at least one redundant drive. It’s the cheapest way to avoid catastrophic data loss.

2. Install Ubuntu (or Debian‑based distro) with Dual‑Boot for Safety

1. Create bootable USB – Use Rufus or dd.
2. Partitioning – Allocate ~300 GB for /home, leaving the rest for OS and swap.
3. Install Ubuntu 24.04 LTS (or your choice).
4. Disable Secure Boot if you plan to use custom kernels or drivers that aren’t signed.

Why dual‑boot? A Windows or another Linux install gives you a fallback if the primary OS becomes corrupted.

3. Install NVIDIA Drivers & CUDA Toolkit

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo ubuntu-drivers autoinstall   # installs the recommended driver (e.g., 535)

Verify with nvidia-smi.

If you’ll use Docker containers that need GPU access, install the NVIDIA Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt update && sudo apt install -y nvidia-docker2
sudo systemctl restart docker

Pro tip: Keep the driver version aligned with your CUDA toolkit to avoid runtime errors.

4. Set Up LM Studio (or JupyterLab) for Development

python3 -m venv ~/ai-env
source ~/ai-env/bin/activate
pip install lmstudio==0.x   # replace with latest

Configure GPU usage in your scripts:

device = torch.device("cuda").

Why LM Studio? It bundles a lightweight IDE, GPU monitoring, and quick access to popular models—perfect for rapid prototyping.

5. Deploy Golem (or Similar Distributed Training Framework)

pip install golem

Configure your node to point to the /home partition where data lives.

Set up a task queue so each job writes checkpoints back to the backup drive, and monitor with golem status.

Why Golem? It turns idle GPUs into compute resources while keeping your local data isolated on the main SSD.

6. Automate Backups: The Core of the Guide

A. Pick a Backup Tool – BorgBackup

| Tool | Pros | Cons | — |------|------|------| — | rsync + cron | Simple, no extra deps. | Manual incremental config. | — | BorgBackup (borg) | Deduplication, compression, encryption. | Slight learning curve. | — | Restic | Fast, easy setup, cloud backends. | No native deduplication. |

We’ll use BorgBackup for its balance of speed and space efficiency.

B. Install Borg

sudo apt install borgbackup

C. Create a Backup Repository

Mount your secondary SATA SSD at /mnt/backup and run:

mkdir -p /mnt/backup/borg_repo
borg init --encryption=repokey /mnt/backup/borg_repo

You’ll be prompted for a passphrase—keep it safe but accessible.

D. Backup Script

Create /usr/local/bin/ai_backup.sh:

#!/bin/bash
# AI Backup Script – Borg + cron

export BORG_REPO=/mnt/backup/borg_repo
export BORG_PASSPHRASE="YOUR_PASS_PHRASE"

SOURCE="/home"
EXCLUDE=("--exclude=.cache" "--exclude=*.tmp" "--exclude=~/datasets/tmp")

borg create \
  --verbose \
  --filter AME \
  --list \
  --stats \
  --compression lz4 \
  "${BORG_REPO}::$(date +%Y-%m-%d-%H%M%S)" \
  $SOURCE ${EXCLUDE[@]}

# Prune older backups (keep last 7 daily, 4 weekly)
borg prune \
  --list \
  -v \
  --keep-daily=7 \
  --keep-weekly=4 \
  $BORG_REPO

Make it executable:

sudo chmod +x /usr/local/bin/ai_backup.sh

E. Schedule with cron

Edit your crontab:

# AI backup – every day at 2 AM
0 2 * * * /usr/local/bin/ai_backup.sh >> ~/ai-backup.log 2>&1

Now backups run unattended. Check ~/ai-backup.log for any errors.

F. Test Restoration

borg extract --verbose \
  $BORG_REPO::2025-12-01-020000 \
  /home/user/test_file.txt

Confirm the file matches the original.

7. Optional: Off‑Site Cloud Backup (Backblaze B2)

1. Create a bucket on Backblaze B2.
2. Install b2 CLI and authenticate.
3. Add to your script after the local backup:

# Upload repo snapshot to B2
b2 upload-file $BORG_REPO /path/to/repo b2://my-ai-backups/$(date +%Y-%m)

This adds an extra layer of protection against hardware failure.

Recap & Call to Action

You now have a complete, automated backup pipeline that protects your AI projects on Linux—from the initial $800 build to nightly snapshots that keep data safe.

What’s your setup? Drop a comment below with your hardware choices, any tweaks you made, or questions you have. And if you’re looking for a GPU upgrade or faster storage, check out our affiliate links:

• – great performance for under $300
• – fast, reliable storage for your OS and datasets

Happy training—and stay backed up!