I have a backup NAS that's normally powered off, but it's scheduled to turn on, perform backup, shut down.
It doesn't wake on LAN and there should be no way of knowing it exists outside of checking DHCP static addresses reservations - and now that I mention it, maybe I should remove it from there too.
This minimises the size of the window, and network-snoopable information, required to compromise this set of backups.
I think, perhaps ignorantly, That may prevent some human being or intelligent agent specifically targeting your nas.
I don't think it would help against situations where your primary system is being encrypted for a while, and thus your backups eventually get overwritten with bad stuff.
The data being backed up is in tiers of importance or 'frequency of change', and based on this the backups are staggered, some daily, some weekly.
I wouldn't often go a full week without checking some file or other, so I think I'd know pretty swiftly if I got infected with an encrypting ransomware virus - hopefully quickly enough to minimise damage.
I also do off-site backups on occasion, so I could roll back to the last full set of off-site backups - so long as I remember where/when the most recent one is.
My main adversary I'm protecting against is hardware failure, however. I think I run a pretty tight ship in restricting access to data that I'd rather not lose.
Maybe some hubris payback will come visit one day though...
>The data being backed up is in tiers of importance or 'frequency of change', and based on this the backups are staggered, some daily, some weekly.
This right here is the key to backups: triage your data to understand what is truly important. I roughly put things into three buckets
- Priority 1 - potentially disastrous if lost. Financials, taxes, legal documents, and password vaults. The nice thing about most of these documents is that they are immutable and likely append-only (eg you only have one set of 2020 taxes). For most people, this amount of data should be well under 1GB and require only sporadic backups. Which means you can purchase a bolus of $10 thumb drives, encrypt the collection, and leave them everywhere. Mail an annual copy to mom, leave one in your bag, at the office -wherever.
- Priority 2 - anything you created which does not fall into Priority 1. Home pictures, videos, your 1000 half-baked programming projects, etc. Potentially a much larger collection for which a real backup system becomes necessary.
- Priority 3 - everything else which is theoretically replaceable. The archive of music you "acquired", backups of youtube videos, personally ripped DVD collection, etc
It's a bit complex and almost always in the midst of a structural change due to new learnings[0][1][2], so describing it in text will be a bit difficult both to read and write. I am planning to do a nice write-up with diagrams once I get myself a proper platform to broadcast myself (which it's taken me ten years to continue to fail to do, so don't hold your breath).
Firstly, as fbdab103 said:
This right here is the key to backups: triage your data to understand what is truly important.
Know your data, know how often it changes, know its level of importance to you, know what duration of loss you're willing to incur (or are likely to incur based on hardware cost limitations).
Hardware:
- Primary NAS with large internal storage and matched external USB storage
- Secondary NAS with same storage size as Primary NAS
- 2x Servers running Proxmox, each with internal storage enough for their VMs data
- Spare older hardware from past upgrades (not live, but as redundancy in case of disaster)
Software:
- Proxmox
- Syncthing
- rsync
- NAS software (QNAP, Synology, TrueNAS, or whatever else)
Configuration:
I have a central store of documents on the Primary NAS, protected by username / password access. Each family member has their own directory.
Mobiles backup photos and videos to a temporary / throw away location on the Primary NAS via a Syncthing docker instance hosted on one of the VMs. This destination is then backed up (twice?) daily using a couple of rsync scripts to separate photos and videos into a permanent location on the Primary NAS.
The secondary NAS has a daily wake up and backup of these 'home directories', photos and videos to both its internal and external storage.
Each VM is hosted on the Proxmox server's internal storage (I used to have all the VM disks hosted on the NAS and connected to Proxmox via ISCSI, but the centralisation to the NAS became an issue when the NAS was lost - see New Learnings 1[0]).
Each Proxmox Server has a weekly scheduled backup of all the VMs to both the Primary and Secondary NAS's, and at least 4 historic backups, in addition to the current, are kept, which Proxmox is configured to manage.
(This is actually against what I said earlier about the Secondary NAS only 'fetching' backups - that's how it used to do it, but this went against the 'purity' of having a direct backup raher than a copy of a backup, which is its own interesting topic: a direct backup is more trustworthy than a copy of a backup because there's less opportunity for data corruption, so the theory goes).
The primary NAS has a weekly backup of these VM backups from its internal storage to external storage.
The secondary NAS has a weekly wake up and backup of these VM backups from its internal storage to external storage.
The secondary NAS also has a weekly wake up and backup of other media stored on the primary NAS, which gets copied to both internal and external storage.
The external storage of either or both of the NAS's can be disconnected and stored elsewhere if I go on holiday, then retrieved and reconnected upon return.
In addition, I do ad-hoc backups to older HDDs and SSDs of different tiers of data, where Family Photos and Videos are probably the most import sentimentally, and financial / household-management documents are the most important practically. These get spread amongst friends and family and are poorly documented (which I need to improve, lest they find said 'spare' hard drive or USB and decide to use it for something far less important).
(I have also recently setup backups for docker instance configuration data, but they're a bit redundant given that these are encapsulated by the VM backups, but it helps me sleep easier to have them backed up separately - these are a weekly script, similarly copied weekly from Primary to Secondary NAS internal and external storage)
[0]: New learnings 1: In mid-August last year my primary NAS became unavailable suddenly and irretrievably with no notice - an incident I'll be documenting in detail some time in the future
[1]: New learnings 2: Due to some power issues late December 2022 I'm rearranging the setup to ride through such things a bit better. This doesn't directly to the backups, but some of the hardware / configuration setup has changed, which has a bearing on the backup configuration.
[2]: New learnings 3: Stagger some of the backups in case of source getting corrupted / encrypted maliciously (may require a manual backup for the final step for a 'known good' backup to a more-secure destination)*
> and thus your backups eventually get overwritten with bad stuff.
I use rsync's --link-dest to get Time Machine-like backups so they can't be overwritten through the backup system itself. I don't take the "shut it down" precautions with this system the other person does, though I do have a separate in-place one that's strictly manual (as in I have to physically plug it in to update).
I would agree, but I feel the GP specifically talked about "Backup NAS", which is where my question pertains.
I do have a set of hard drives I exchange with my friend as "off site" backup. Neither my cloud service nor NAS keep copies/snapshots of previous versions of files; and I have too much stuff to create BlueRay let alone DVD one-off backups :0/
If your main computer has had all its files encrypted by ransomware, will the backup NAS know not to replace the good backup with a bad one? i.e. hopefully it's not doing something like `rsync --delete`.
That's not hard. On my backup server, I have an account with a restricted shell that my backed-up machine has an ssh key for. That restricted shell is simply the following script:
#!/bin/bash
cd ~/backups
tar --restrict --keep-old-files -x
So the only thing that the backed-up machine (or an attacker) can do with the ssh key is push new files onto the backup server as a tar stream - it can't overwrite any files, and it can't put any files anywhere except the correct directory.
For my server I went one step further and set it up so that the server being backed up has no access to the backup system at all and insteady only prepares encrypted (differential) tar archives that the backup system then grabs.
Restic is my current backup solution of choice and that takes snapshots by default. If you took a backup on Monday, ransomwared on Tuesday, your backup volume would double in size (file de-duplication is going to fail spectacularly), but the Monday backup will be unimpacted until a prune operation is run. Suggested prune workflow is to maintain fairly staggered snapshots eg: 1x six-months ago, 1x three-months ago, 3x from last month, whatever makes you comfortable. Which should give a pretty comfortable margin on retaining your files.
It doesn't wake on LAN and there should be no way of knowing it exists outside of checking DHCP static addresses reservations - and now that I mention it, maybe I should remove it from there too.
This minimises the size of the window, and network-snoopable information, required to compromise this set of backups.