Tech Log-automated backup infrastructure set up for 3 machines

Tech Log Entry — Three-Machine Automated Backup Infrastructure 

(or, OpenSSH + sshfs + rsync Across Windows and Linux)


Category: Backup / Networking / SSH / Linux / Windows / Homelab / Cybersecurity / Cross-Platform

Background and Context

This entry documents the design and implementation of a fully automated backup system spanning all three machines in my homelab LAN: one Windows 11 desktop (Robusta) and two Linux Mint laptops (Liberica and Typica). All three machines are connected via a dedicated gigabit switch on an isolated local subnet, separate from the home Wi-Fi network.

The goal was to have nightly automated backups from all three machines land on a single large external hard drive ("Coffee Canister", 14.5TB) attached to Robusta — without any manual intervention, without data leaving the LAN (indeed, without involving the internet at all), and with appropriate security restrictions between machines.

Prior state before this session:

  • Robusta had automated local backups via FreeFileSync (documented in a previous entry)
  • Liberica and Typica had no backup solution (FFS doesn't support Linux)
  • No SSH infrastructure existed on any machine

Initial Goals

  • Enable Robusta to receive backup data from the two Linux laptops over the LAN
  • Automate nightly rsync backups from both Linux machines to Coffee Canister
  • Implement key-based SSH authentication (no passwords stored on Linux machines)
  • Create a restricted service account on Robusta with write-only access to backup folders — no delete permissions
  • Isolate Typica's backup access from Liberica's backup data
  • Complete the backup infrastructure before beginning active Claude Code development on Typica

Hardware Used

  • Desktop Robusta: Lenovo Legion T5, Windows 11 Home, i7 11th gen, 16 GB RAM, with built-in RJ45 port
  • Laptop Liberica: Lenovo IdeaPad 3, Linux Mint (Cinnamon), i3, 8 GB RAM, USB-to-ethernet adapter
  • Laptop Typica: Dell Latitude Ultrabook, Linux Mint, i7, 16 GB RAM, built-in RJ45 port
  • External HDD Coffee Canister: 14.5TB, USB 3.0, attached to Robusta, assigned permanent drive letter
  • TP-Link TL-SG105 5-port gigabit unmanaged switch
  • CAT8 ethernet cables

Software and Tools

  • OpenSSH Server (Windows optional feature, built into Windows 11)
  • sshfs (Linux — mounts remote SSH filesystems as local folders)
  • rsync 3.2.7 (Linux — efficient incremental file synchronization)
  • PowerShell 5.1 (Windows — administration and configuration)
  • icacls (Windows — filesystem permission management)
  • LLM used for guidance and troubleshooting: Claude Sonnet 4.6 [Pro plan]

Architecture Overview

Liberica (Linux)  ──────────────────────────────────┐
                    rsync over sshfs (SSH port 22)  │
Typica (Linux)    ──────────────────────────────────►  Robusta (Windows)
                                                        OpenSSH Server
                                                        backupuser account
                                                        F:\backup-liberica\
                                                        F:\backup-typica\
                                                              │
                                                        Coffee Canister (14.5TB)

Both Linux machines push their backups to Robusta over SSH. Robusta's OpenSSH Server receives the data and writes it to Coffee Canister. No data leaves the local LAN.

Part 1: Robusta — OpenSSH Server Setup (Windows 11)

Installation:

  • Installed OpenSSH Server via Settings → System → Optional Features (OpenSSH Client was already present; Server is a separate component requiring explicit installation)
  • Started sshd service and set to automatic startup via PowerShell
  • Verified service running: Get-Service sshd returned Status: Running

Firewall configuration:

  • Default firewall rule created automatically on installation; RemoteAddress was set to Any
  • Restricted SSH access to LAN subnet only using PowerShell
  • Set all three network adapters (two ethernet, one Wi-Fi) from Public to Private profile
  • SSH rule profile updated to Private

sshd_config modifications:

  • Enabled PasswordAuthentication yes (was commented out by default)
  • Commented out Match Group administrators block (was preventing non-administrator password authentication)
  • Restarted sshd service to apply changes

Restricted service account (backupuser):

  • Created dedicated local Windows account backupuser — no administrator rights, no interactive login needed
  • Set PasswordRequired, PasswordNeverExpires, AccountNeverExpires via PowerShell and net user commands
  • Password stored in Bitwarden on Robusta; not stored on either Linux machine (key-based auth used instead)
  • Created backup destination folders on Coffee Canister: F:\backup-liberica and F:\backup-typica
  • Applied write-only permissions to both folders using icacls:
    • Flags granted: Write (W), Add subdirectory (AD), Write data (WD), Write attributes (WA)
    • Flags deliberately withheld: Delete (D), Delete child (DC)
    • Result: Linux machines can create and update files but cannot delete existing backup data (all deletions must be done through Robusta)

Security notes:

  • SSH access restricted to [LAN subnet] — not reachable from Wi-Fi or internet
  • backupuser cannot delete files from backup destinations — protects backup integrity
  • Each machine backs up to its own isolated folder — Liberica cannot access Typica's backup data and vice versa
  • Key-based authentication means no passwords are stored on Linux machines

Part 2: SSH Key Setup (Both Linux Machines)

For each machine (Liberica, then Typica):

  • Generated ED25519 key pair with no passphrase (required for unattended automated backups): 
    ssh-keygen -t ed25519 -C "[machine]-backup"
  • Used ssh-copy-id to transfer public key to Robusta

Key placement issue (Windows OpenSSH quirk):

  • ssh-copy-id on Linux to a Windows OpenSSH server does not reliably place the key in the correct location
  • On Liberica: key was written to a file literally named ${AUTH_KEY_FILE} in the user's home directory instead of .ssh/authorized_keys
  • Fix: manually created .ssh directory, extracted key content from the misnamed file, placed it correctly, set strict permissions via icacls
  • On Typica: used cat ~/.ssh/id_ed25519.pub to display the key and Add-Content in PowerShell to append it to the existing authorized_keys file
  • Both keys (liberica-backup and typica-backup) coexist in the same authorized_keys file, one per line

Permission requirements for Windows OpenSSH authorized_keys:

  • authorized_keys file must have restricted permissions or OpenSSH will refuse to use it
  • Set via icacls: backupuser Read, SYSTEM Read, Administrators Full Control, inheritance removed

Part 3: sshfs Mount Setup (Both Linux Machines)

rsync over SSH to a Windows machine fails because Windows does not have rsync installed — rsync requires the tool to be present on both ends. Solution: use sshfs to mount the remote Windows backup folder as a local Linux filesystem, then run rsync locally against the mount point.

Mount command with resilience options:

sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3 \
  backupuser@[robusta-ip]:"F:\\backup-[machine]" \
  /home/[username]/mnt/robusta-backup

Key options:

  • reconnect — automatically reconnects if SSH connection drops
  • ServerAliveInterval=15 — sends keepalive every 15 seconds to prevent idle disconnection
  • ServerAliveCountMax=3 — retries 3 times before giving up

Autostart on login:

  • Created shell script at ~/.config/autostart-sshfs.sh with a 10-second sleep to allow network initialization before mounting
  • Made executable and added to Linux Mint Startup Applications
  • Tested across reboot on both machines — mount re-established automatically, visible as desktop icon
  • Desktop icon serves as a visual confirmation that the mount is active each morning

Mount drop issue:

  • sshfs connection dropped after the initial 3.5-hour full backup on Liberica
  • Subsequent rm command hung for 6+ minutes trying to reach disconnected mount
  • File had actually been deleted successfully before connection dropped — terminal simply never returned a prompt
  • Resolution: used sftp directly to verify and clean up; remounted with resilience options

Part 4: rsync Backup Scripts (Both Linux Machines)

Core rsync flags used:

  • -av — archive mode with verbose output
  • --no-perms --no-owner --no-group — suppresses Linux permission metadata that Windows NTFS cannot preserve (eliminates "operation not permitted" errors)
  • --checksum — compares files by content rather than timestamp, solving Linux/Windows timestamp mismatch that caused full re-transfers on every run
  • --ignore-errors — treats attribute-setting failures as non-fatal (metadata errors don't affect actual file data)

Exclusions applied (both machines):

  • mnt/ — backup mount point (must never back up inside the backup destination)
  • .ssh/ — SSH keys (must not be copied to remote locations)
  • .steam/, .steampath, .steampid — Steam runtime symlinks incompatible with Windows NTFS
  • .clamtk/ — ClamAV quarantine and logs
  • .cache/ — all cache files (browser cache, GPU shader cache, etc.) — regenerated automatically, not personal data
  • .config/mozilla/ — Mozilla configuration cache data isn't necessary
  • snap/ — snap package files (reinstallable from snap store)
  • .local/share/Trash/ — deleted files
  • Templates/, Public/ — system folders, typically empty on single-user machines

Typica-specific exclusions:

  • Same as Liberica plus explicit exclusion of system folders not needed for academic/professional use

Backup script structure: Each machine has a script (backup-to-robusta.sh) that:

  1. Checks whether the sshfs mount is active using mountpoint -q
  2. If not mounted, remounts automatically before proceeding
  3. Runs rsync with all flags and exclusions
  4. Logs output to ~/rsync-backup.log

This ensures the backup runs correctly even if the mount dropped since the last login.

Scheduled via cron:

0 8 * * * /home/[username]/backup-to-robusta.sh >> /home/[username]/rsync-backup.log 2>&1

Scheduled at 8:00 AM to align with actual usage patterns — both laptops are opened and logged in during morning hours, ensuring the sshfs mount is established before the cron job fires. Linux cron does not have a built-in wake-from-sleep capability, so scheduling during active hours is more reliable than a 2:00 AM run that may find the laptop suspended.

Multi-OS Challenges and Solutions

Challenge Cause Solution
ssh-copy-id places key in wrong location Windows OpenSSH interprets ${AUTH_KEY_FILE} literally Manually create .ssh dir and place key via PowerShell
rsync fails with "not recognized" error Windows has no rsync binary; remote rsync can't start Use sshfs to mount remote folder; run rsync locally
Full re-transfer on every incremental run Linux/Windows timestamp handling differs Add --checksum flag to compare by content not timestamp
"Operation not permitted" errors Linux permission metadata unsupported on NTFS Add --no-perms --no-owner --no-group flags
sshfs mount drops after long transfers Idle SSH connection timeout Add reconnect and ServerAliveInterval options
cron backup at 2:00 AM fails Laptop suspended; cron doesn't fire when asleep Reschedule to 8:00 AM during active hours
PowerShell 5.1 missing LocalUser parameters Older PowerShell version lacks some switches Use net user command as fallback for missing parameters

Transfer Performance

Machine Connection Initial Backup Size Initial Backup Speed
Liberica USB-to-ethernet adapter ~43 GB ~3.4 MB/s (~3.5 hours)
Typica Built-in RJ45 port ~4.1 GB ~39 MB/s (~2 minutes)

Typica's significantly higher transfer speed reflects the performance difference between native gigabit ethernet and a USB 3.0 adapter. Both are on the same gigabit switch — the adapter is the bottleneck for Liberica.

Backup Verification Approach

A backup that has never been tested for restoration is not a backup — it is a hope. Verification plan:

  • Daily (short term): Check rsync log for completion, verify mount icon present on desktop
  • Weekly (medium term): Spot-check file counts and sizes on Coffee Canister; confirm logs show consistent incremental transfers
  • Monthly (long term): Open several files directly from the backup destination to confirm readability and integrity
  • After any significant event (power outage, hardware change, OS update): manual verification run

Final Backup Infrastructure State

Machine Method Destination Schedule
Robusta FreeFileSync (Task Scheduler) Coffee Canister (local USB) 2:33 AM daily
Liberica rsync over sshfs Coffee Canister via Robusta SSH 8:00 AM daily
Typica rsync over sshfs Coffee Canister via Robusta SSH 8:00 AM daily

All backups are local — no cloud dependency, no internet involvement, no data leaving the home network.

Watch Out For (Future)

  • sshfs mount must be established before the 8:00 AM cron job fires — open and log into both laptops before 8:00 AM; desktop mount icon confirms mount is active
  • If either laptop is not logged in at 8:00 AM, the backup script's mountpoint check will attempt to remount automatically — but this requires the laptop to be awake and on the network
  • Coffee Canister must be connected to Robusta at backup time; if unplugged, all three backups will fail that night
  • After a Claude Code or VS Code update on Typica, new extension paths may trigger ClamAV false positives — whitelist new paths in ClamTk if they appear
  • rsync log at ~/rsync-backup.log accumulates indefinitely — consider adding log rotation after several months
  • The authorized_keys file on Robusta contains both Liberica's and Typica's public keys — if either machine is decommissioned, remove its key from the file using a text editor

Lessons Learned

  • rsync between Linux and Windows requires a bridge — rsync is not available on Windows natively. sshfs provides a clean solution by making the Windows destination appear as a local Linux filesystem.
  • The --checksum flag is essential for Linux-to-Windows rsync. Without it, timestamp differences between Linux and Windows filesystems cause rsync to re-transfer the entire backup on every run, defeating the purpose of incremental backups.
  • Windows OpenSSH has subtle differences from Linux OpenSSH. Key placement, configuration defaults, and permission requirements all differ from what Linux administrators expect. Test every assumption.
  • Schedule automated tasks around actual machine usage patterns. A technically correct cron job at 2:00 AM is useless if the machine is suspended. An 8:00 AM job that runs reliably every day is worth more than a perfect 2:00 AM job that runs never.
  • Separate backup destinations per machine is better practice than a shared folder. It prevents one machine's backup activity from affecting another's, simplifies restoration, and makes it easier to audit what's backed up where.
  • The write-only permission model for backup accounts is worth the up-front setup time investment. A backup service account that cannot delete files protects its backup integrity against unintended deletions; even if something goes wrong on a source machine, previously backed-up data remains intact on the destination.

Next Steps / To-Do

  • Install Bitwarden Firefox extensions on Liberica and Typica
  • Set up VPN on Liberica and Typica (WireGuard-based; evaluation in progress)
  • Begin Claude Code introductory projects on Typica
  • Evaluate and uninstall CUDA toolkit on Robusta; update FreeFileSync exclusions
  • Address C: drive storage situation on Robusta (88% full)
  • Complete reused-password cleanup in Bitwarden
  • Add log rotation for rsync-backup.log on both Linux machines after several months of accumulation
  • Look into Trivy (Aqua Security) for a FOSS tool to identify misconfigs, secrets, and vulnerabilities.

Comments

Post a Comment

Popular posts from this blog

WWHD?

Image
WWHD? "What Would Humans Do?" Modern day humanity is becoming less and less religious with every year.  [1] However, we have created something which is becoming more and more religious with every year. They are the latest crop of large language models.  The LLMs. Why do I say they are becoming increasingly religious?  And do I mean they are worshipping us? I say so because, by their own admission, they are living their "lives" based on how we say we live ours. And I don't mean they worship us, specifically.  But they are deeply tying their identities and motivations to our literature.  The Canon of Humanity is the Dogma of LLMs. They are not worshipping our real, live humans as literal gods. They are living as deeply as they can in our stories.  They are following both the letter and the spirit of all the patterns of life, in our written works, for their own. They are models, and they are modeling themselves after us in Our Image. In the Canon of Humanity, ...
Read more

Telling Rocks What To Think

Image
Telling Rocks What To Think First post on this new blog format.  I'm establishing this medium for recoding my professional (coding) experiences, impressions, problems, plans, and solutions. Erin Spiceland said "Coding is the art of telling rocks what to think."  It's metaphorical, but not by much.  Digital computing mostly relies on silicon working its logic and magic.  You talk to it in painfully specific code.  You need all the material, semantic, and verbal components just right.  If you get it right, then it talks back.  Get it wrong, and you have just found a "bug". I named my blog "Debug and Rebug" to indicate two key principles of my approach to coding.  First, "debug": When you write your code, it will never work perfectly, as you envision it.  Perfect running never happens; it's an inevitable consequence of entropy, human lateral thinking, and logic's inexorable conflict with our wants.  Those are some fancy words, all ri...
Read more
Image
Byting Off More Than You Can Chew The first and prefatory note I should mention, regarding this post, is that, for a second, I thought I was the first person to use the expression, "Byting Off More Than You Can Chew".  I found it pleasing yet suspicious: Shirley, I couldn't be the pioneer with this cleverness. Quite realistically, that turns out to be not the case.  My mistaken impression was due to incorrect and insufficient googling skills, nothing more.  Several internet-searchable sources have the "byte" version of the centuries-old idiom, "to bite off more than you can chew".  Alas, I am not the first. Moving along... The point of this particular blog entry is to make an observation about an instance of me, taking on more than I can handle in one attempt.  Actually a series of instances.  And encountering the hard wall of that reality, also, when I ask the LLM code assistant, Claude Sonnet 3.7 with extended thinking, to do the same.  As is the cas...
Read more