I also use restic and do backups to append-only rest-servers in multiple locations.
I also back up multiple hosts to the same repository, which actually results in insane storage space savings. One thing I'm missing though is being able to specify multiple repositories for one snapshot such that I have consistency across the multiple backup locations. For now the snapshots just have different ids.
I once met the Borg author at a conference, pretty chill guy. He said that when people file bugs because of data corruption, it's because his tool found the underlying disk to be broken. Sounds quite reliable although I'm mostly fine with tar...
I used CrashPlan in 2014. Back then, their implementation of Windows's Volume Shadow Copy Service (VSS) was buggy, and I lost data because of that. I doubt my underlying disk was broken.
Restic is far better both in terms of usability and packaging (borgmatic pretty much is a requirement for usability). Have used both extensively, you can argue that borg can just be scripted instead and is a lot more versitile, but I had a much better experience with restic in terms of setup and forget. I am not scared that restic will break, with borg I did.
Also not sure why this was posted, did a new version release or something?
And that's what I did myself. Organically it grew to ~200 lines, but it sits in the background (created a systemd unit for it, too) and does its job. I also use rclone to store the encrypted backups in an AWS S3 bucket
I so much forget about it that sometimes I have to remind myself to test it out if it still works (it does).
Original size Compressed size Deduplicated size
All archives: 2.20 TB 1.49 TB 52.97 GB
Depends on what you consider large; I looked at one of the machines (at random), and it backups about two terabytes of data spread across about a million files. Most of them aren't changing day to day. I ran another backup, and restic rescanned them & created a snapshot in exactly 35 seconds, using ~800 MiB of RAM at peak and about 600 on average.
The files are on HDD, and the machine doesn't have a lot of RAM, looking at high I/O wait times and low CPU load overall, I'm pretty sure the bottleneck is in loading filesystem metadata off disk.
I wouldn't backup billions of files or petabytes of data with either restic or borg; stick to ZFS for anything of this scale.
I don't remember what the initial scan time was (it was many years ago), but it wasn't unreasonable — pretty sure the bottleneck also was in disk I/O.
Last time I checked the deduplication only works per host when backups are encrypted, which makes sense. Anyway, borg is one of the three backup systems I use, it's alright.
I’ve been looking at this project occasionally for more than four years. The development of version 2.0 started sometime in April 2022 (IIRC) and there’s still no release candidate yet. I’m guessing that it’ll be finished in a year from now.
What are the current recommendations here to do periodic backups of a NAS with lower (not lowest) costs for about 1 TB of data (mostly personal photos and videos), ease of use and robustness that one can depend on (I know this sounds like a “pick two” situation)? I also want the backup to be completely private.
You definitely should have checksumming in some form, even if compression and deduplication are worthless in this particular use case, so either use ZFS on both the sending and the receiving side (most efficient, but probably will force you to redo the NAS), or stick to restic.
I've been mostly using restic over the past five years to backup two dozen servers + several desktops (one of them Windows), no problems so far, and it's been very stable in both senses of the word (absence of bugs & unchanging API — both "technical" and "user-facing").
The important thing is to run periodic scrubs with full data read to check that your data can actually be restored (I do it once a week; once a month is probably the upper limit).
restic check --read-data ...
Some suggestions for the receiver unless you want to go for your own hardware:
I've had an awful experience with Duplicati. Unstable, incomplete, hell to install natively on Linux. This was 5 years ago and development in Duplicati seemed slow back then. Not sure how the situation is now.
I switched to restic (https://restic.net/) and the backrest webui (https://github.com/garethgeorge/backrest) for Windows support. Files are deduplicated across machines with good compression support.
One big advantage of using restic is that its append only storage actually works unlike for Borg where it is just a hack.
I also use restic and do backups to append-only rest-servers in multiple locations.
I also back up multiple hosts to the same repository, which actually results in insane storage space savings. One thing I'm missing though is being able to specify multiple repositories for one snapshot such that I have consistency across the multiple backup locations. For now the snapshots just have different ids.
I once met the Borg author at a conference, pretty chill guy. He said that when people file bugs because of data corruption, it's because his tool found the underlying disk to be broken. Sounds quite reliable although I'm mostly fine with tar...
I used to work on backup software. I lost count of the number of times this happened to us with our clients too
I used CrashPlan in 2014. Back then, their implementation of Windows's Volume Shadow Copy Service (VSS) was buggy, and I lost data because of that. I doubt my underlying disk was broken.
Restic is far better both in terms of usability and packaging (borgmatic pretty much is a requirement for usability). Have used both extensively, you can argue that borg can just be scripted instead and is a lot more versitile, but I had a much better experience with restic in terms of setup and forget. I am not scared that restic will break, with borg I did.
Also not sure why this was posted, did a new version release or something?
> you can argue that borg can just be scripted
And that's what I did myself. Organically it grew to ~200 lines, but it sits in the background (created a systemd unit for it, too) and does its job. I also use rclone to store the encrypted backups in an AWS S3 bucket
I so much forget about it that sometimes I have to remind myself to test it out if it still works (it does).
How is the performance for both?
Last time I used restic a few years ago, it choked on not so large data set with high memory usage. I read Borg doesn't choke like that.
Depends on what you consider large; I looked at one of the machines (at random), and it backups about two terabytes of data spread across about a million files. Most of them aren't changing day to day. I ran another backup, and restic rescanned them & created a snapshot in exactly 35 seconds, using ~800 MiB of RAM at peak and about 600 on average.
The files are on HDD, and the machine doesn't have a lot of RAM, looking at high I/O wait times and low CPU load overall, I'm pretty sure the bottleneck is in loading filesystem metadata off disk.
I wouldn't backup billions of files or petabytes of data with either restic or borg; stick to ZFS for anything of this scale.
I don't remember what the initial scan time was (it was many years ago), but it wasn't unreasonable — pretty sure the bottleneck also was in disk I/O.
Love borg, use it to backup all my servers and laptop to a Hetzner Storage Box. Always impressed with the deduplication stats!
Same story here, using Borg with a Hetzner storage box to give me offsite backups.
Cheap, reliable, and almost trouble-free.
Last time I checked the deduplication only works per host when backups are encrypted, which makes sense. Anyway, borg is one of the three backup systems I use, it's alright.
Which are the others?
https://kopia.io/
backuppc and a shell script using rsync, for backups to usb sticks
I using it with via Vorta (https://vorta.borgbase.com) frontend. My favorite backup solution so far.
Pika Backup (https://apps.gnome.org/PikaBackup/) pointed at https://borgbase.com is my choice.
I’ve been looking at this project occasionally for more than four years. The development of version 2.0 started sometime in April 2022 (IIRC) and there’s still no release candidate yet. I’m guessing that it’ll be finished in a year from now.
What are the current recommendations here to do periodic backups of a NAS with lower (not lowest) costs for about 1 TB of data (mostly personal photos and videos), ease of use and robustness that one can depend on (I know this sounds like a “pick two” situation)? I also want the backup to be completely private.
You definitely should have checksumming in some form, even if compression and deduplication are worthless in this particular use case, so either use ZFS on both the sending and the receiving side (most efficient, but probably will force you to redo the NAS), or stick to restic.
I've been mostly using restic over the past five years to backup two dozen servers + several desktops (one of them Windows), no problems so far, and it's been very stable in both senses of the word (absence of bugs & unchanging API — both "technical" and "user-facing").
https://github.com/restic/restic
The important thing is to run periodic scrubs with full data read to check that your data can actually be restored (I do it once a week; once a month is probably the upper limit).
Some suggestions for the receiver unless you want to go for your own hardware:https://www.rsync.net/signup/order.html?code=experts
https://www.borgbase.com
(the code is NOT a referral, it's their own internal thingy that cuts the price in half)
I've been using it for ~10 years at work and at home. Fantastic software.
I remember using Borg Backup before eventually switching to Duplicati. It's been a while.
I currently use borg, and have never heard of Duplicati. What made you switch?
I've had an awful experience with Duplicati. Unstable, incomplete, hell to install natively on Linux. This was 5 years ago and development in Duplicati seemed slow back then. Not sure how the situation is now.
Interesting to hear. I use Duplicati on Windows and it's been fine, though I haven't extensively used its features.