I'm really good at tangents. Not the line-intersects-circle type (particularly), but the here-I-am-working-on-my-favorite-software-project-when-I-start-a-new-one-instead type. For quite a while now, I've been unhappy with the data backup solution I've been using on my servers. Years, really. And recently I added a Windows server, which just made it worse. Windows has a way of doing that. In any case, the problem really isn't one of platform, or software usability or performance. It's about horizon and rotation. The problem is philosophy.
Once upon a time
In the olden days, when men were real men and used tape cartridges for backup (some with as much as 4GB of storage), we would do "tape rotation." The mainframe folks, being the lazy sots there always were, had a robot do it for them, but that's another tangent. The simplest rotation scheme looks like this:
Simple, sure, but since you only have the one tape, if it goes south, so does your data. And once it fills up, you have an even bigger problem: erase and start over, or get another tape. That's a biggie. So then we did "round-robin" wherein we used a different tape for each day of the week. Wow. Progress. Same problem, only it took longer to realize.
The amount of time you can keep a copy of data on a backup media is called the "Backup Horizon" and is perhaps the most important and yet least designed-for attribute of backup solutions through the ages. Attempts were made, to be sure, but they had to deal with the fact that tapes cost money, often a lot of money. Somebody at some point came up with the "father-son" rotation strategy, wherein you used four tapes, one for each of Monday - Thursday (the world was simpler when nobody worked on weekends) and four more tapes, one for each Friday in the month. Each Friday, you'd do a Full backup (all data gets copied to tape), and then Monday thru Thursday you'd do either an Incremental (all files changed or added since the last day's backup go to tape) or a Differential (all files changed or added since the last Full backup go to tape) on each weekday. This extended the apparent (not real, more below) backup horizon to one month (give or take). Adding a grandfather level to this (twelve "monthly" tapes) gives an apparent backup horizon of one year.
Enter reality
But the truth is, it doesn't really work that way, for many important reasons. The first is that data changes more rapidly than the rotation can effectively deal with. If an important file is created on Monday, it gets put on the Monday tape. But then if it's deleted on Tuesday, it's basically forgotten. Let's say you need to recover it from tape on Wednesday: no worries. But try that again the following Tuesday. No joy. Tape's been overwritten. Ok, so maybe you stack your diffs (more than one backup session per tape). When that runs out is a function not of how long you want to retain your data, but how full the tapes get. If you put a no-overwrite policy in place, you end up with a mess of tapes with strange names like 'Monday #4c" which in no way relates to anything meaningful. Many backup plans are not so much data retention plans as disaster recovery plans. If something goes horribly wrong, you get back to almost where you were, but not quite. But hey, that's what disaster means, yeah?
But a bigger issue is that if any one tape goes south, that data is just plain gone. You may or may not have redundant copies somewhere. Good luck finding and verifying them, in any case. And since the more you use a backup tape, the closer you get to tape failure, the system is inadvertently designed to lose exactly the kind of data you will most likely want to recover - the highly volatile, transient stuff. The document you accidentally cut from but forgot to paste back into and didn't notice since you were moving a paragraph from page 10 to page 225.
As disk space began to seriously eclipse tape capacity, and as data became more decentralized (no longer on the corporate servers), these backup solutions started having trouble keeping up. And fell into disuse. Yeah, sure, all the big shops have backup server farms now, but it's not as easy for the smaller shops. So we tend to use the "a copy is a backup" philosophy. And it ain't.
Return of the King (or at least he's started the journey)
The only backup solution I've ever seen that DID get it right was called "Network Archivist" from a now-defunct company called Palindrome. Their offices were only a few miles from my current house, and just down the block from my former data center. Not that that has any relevance, since at the time I was using NA, I lived something like 40 miles away and didn't even have a data center.
So because I need a better backup solution for my own servers and those of some clients, and all the offerings on the market now are of the just-copy-it-somewhere-and-call-it-a-backup variety...
