Backup to Blu-ray

[RustyTheGeek]

I have never bought drives with consecutive serial numbers, I do check to see that they are from different batches, they often have different firmware versions as well, but I do generally buy them from the same manufacturer. My drives are powered up 24/7, they are on a UPS and protected from spikes and brownouts. The raid 5 arrays can withstand 2 drives failing and keep on going. My current set of six 2TB drives are reaching 5 years old, with no failures. I'll likely be replacing them with either 6TB Drives, or SSD's. The NAS before this one lived 5 years then the power supply died, but the disks were not affected. Even before that, I had a 4 drive Raid Disk Array of 150TB Drives, no failures, Before that, 4 - 72 TB Drives, Before that, 4 36 TB and before that a 4 Drive Raid Array of 13GB Disks (1990's) with no failures. All of my CD's made during the late 80's to late 90's rotted away, but I still have that old data on my hard drives.

I think Mt. Spokane is pretty sharp, has some great skill and experience with storage arrays. In short, he gets it, he knows what he's doing and he is covering all the bases. I also think that he is incredibly lucky. RAID 5 has diminished in popularity because it is so fragile and only tolerates ONE drive failure in the array. RAID 6 (among several other types) is becoming a more common PARITY Array type because RAID 6 can tolerate 2 simultaneous drive failures in the given array. RAID 5 performance is fair at best, RAID 6 performance is poor. Rebuilds after a drive failure in either array can take days, weeks or even months. For this reason, RAID 1+0 (10) is also popular if performance is more important. I won't go into the endless details of RAID here but I will say that unless you are ready to face a steep learning curve and a lot of stress and expense, don't implement a RAID array for yourself any more complicated than a RAID 1 mirror. It's just not worth it IMHO. Not only do you need to understand the technology, you need to understand the hardware and how to operate it. And once you venture past a simple RAID 1 mirror array, the hardware is critical for performance, acceptable reliability and even the possibility of recovery.

In my experience, drives fail in an unpredictable way and I'm amazed that Mt. Spokane has had such a long timeline with no failures. So while that is wonderful for him, I think it's a bit outside of the expected norm. I also predict that once he installs 6TB drives, he will probably see his first failure(s). It seems like as drive capacity/density has increased over the last few years, so has the failure rate (or at least the likelihood of data loss). And just because a the drives haven't failed, doesn't mean there can't be data corruption or loss. Backups still must be maintained and like you might expect, the more complicated the array, the easier it is for errors to creep in. Which is why you need more expensive controllers, etc for any array more complex than RAID 1.

BUT - we are digressing. This thread is about BACKUP. A lot of ideas have been tossed around. I stated my thoughts above and I'll repeat that if you put your faith in writable dye based media, you better test it every year or two because there is a definite history of this type of media failing after a few years. As for what kind of drive to use, I won't argue about what drives are best. All I know is that we all have a LOT of data to back up in the TB range. So external hard drives, whatever kind you prefer, are about all that is affordable and fast enough to get the job done. And while 'The Cloud' is big, it is very slow and out of your control. (Hello Mr. Dotcom?)

 

[RustyTheGeek]

LDS - I won't argue with your replies to my points above. I think we are both right and I suspect you probably would agree with that. I'm glad you shared your knowledge because that way, we all learn something!

In general, I think hard drive technology, after all these years/decades, is amazing and impressive but also somewhat disappointing. Because it is a mass produced item that has continued to drop in profit margins it is a dying product and yet is still necessary. So the overall quality is dropping slowly. But I do think that the quality is fairly consistent across all products with there basically being two classes, consumer/retail and enterprise. What you say is true about warranties and enterprise vs consumer lines but in this case, we are asking to copy data to a drive for a few hours or days and then store the drive away. It doesn't need to be a mil-spec 15,000 RPM database grade drive to do that. It just needs to hold the data. If someone is smart, they will check it every couple years and eventually copy the data from it to (hopefully) a better and more reliable media in the not-too-distant future. And that is all we need it to do. Hold the data for a few (5, 10, 15?) years. At least that is my realistic expectation.

 

[epsiloneri]

Wow, that is deep, I nodded off a third of the way down, dropped my blooming ipad.

I hope your ipad wasn't harmed in any way. An interesting thing about the internet is that no matter how fringe your query, there is bound to be a small group somewhere that makes it their life's mission to delve into that topic with excruciating detail. Archiving is such a topic.

I won't go into the endless details of RAID here but I will say that unless you are ready to face a steep learning curve and a lot of stress and expense, don't implement a RAID array for yourself any more complicated than a RAID 1 mirror. It's just not worth it IMHO. Not only do you need to understand the technology, you need to understand the hardware and how to operate it. And once you venture past a simple RAID 1 mirror array, the hardware is critical for performance, acceptable reliability and even the possibility of recovery.

I agree with your points about RAID 5 being inadequate for the big disks of today, and that RAID 6 has slow performance. But RAID 10 is very expensive if you don't need the performance (for 5 or more disks, say) and RAID 6 should be good enough for on-line back-up solutions. Any non-enterprise user should stay with software RAID, so understanding hardware quirks shouldn't be necessary. I thus think that RAID 6 is more often the better option for the usage pattern expected for those readers of this thread with massive backup needs. Better still would be RAID Z2, using the ZFS filesystem that protects the data against corruption through bit rot, but that is more for technically inclined.

It seems like as drive capacity/density has increased over the last few years, so has the failure rate (or at least the likelihood of data loss).

I know there is the perception that this is true, but is it indeed factually true? Hard drive failures have always been an issue, with some notable examples, e.g. the IBM deskstar 75GXP (aka the "deathstar"), and more recently the Seagate 1.5TB Barracuda.

In the end, I think this about the drive quality is a bit of a red herring. Yes, you should avoid obvious duds (such as the drives mentioned above), and yes, enterprise disks have more stringent quality control. But in the long run, all drives, without exception, will fail. The trick is to plan for it, and take appropriate measures.

BUT - we are digressing. This thread is about BACKUP.

While backup is indeed in the title of the thread, I think this thread is actually about data archiving:
The key difference between backup vs. archiving is that data backups are designed for the rapid recovery of operational data, while data archiving stores data that's no longer in day-to-day use but must still be retained.

I stated my thoughts above and I'll repeat that if you put your faith in writable dye based media, you better test it every year or two because there is a definite history of this type of media failing after a few years.

Yes, that is why regular non-LTH blu ray discs are a much better option. Or the (more expensive) m-disc DVDs. They are not based on organic dye, so do not decay on short time scales. LTH BDs use organic dye like regular writable CDs/DVDs, so don't use them for archiving.

 

[Oceo]

I do backups on the "3-2-1" plan, three copies of the file, in two different physical formats, and one copy off-site. For on-site I copy files to independent Hard Drives, one internal, one external. As to OP's question - for my off-site copy I started using 25GiB Blu-ray about a year ago. It still meant too much time waiting for disks to complete writing. After all, a few of us are shooting with 128GiB CF cards, more of us use 64GiB cards, and many more of us use 32GiB cards in our cameras. ( I shoot a total of more than 1TB/year from three cameras.)

I've almost decided to go to LTO-3 tape. Refurbished writers are less than $1K, used but functioning writers about 1/3 of that. Beside the longevity advantages mentioned in previous comments I expect non-attended backup software must be available so I don't have to sit changing cartridges as I now do with Blu-ray disks. (Does anyone know of a consumer-quality Blu-ray writer that takes a stack of blanks and runs unattended?)

I'd appreciate reading more from those respondents who have experience with LTO-n tape drives and media.

 

[RustyTheGeek]

I do backups on the "3-2-1" plan, three copies of the file, in two different physical formats, and one copy off-site. For on-site I copy files to independent Hard Drives, one internal, one external. As to OP's question - for my off-site copy I started using 25GiB Blu-ray about a year ago. It still meant too much time waiting for disks to complete writing. After all, a few of us are shooting with 128GiB CF cards, more of us use 64GiB cards, and many more of us use 32GiB cards in our cameras. ( I shoot a total of more than 1TB/year from three cameras.)

I've almost decided to go to LTO-3 tape. Refurbished writers are less than $1K, used but functioning writers about 1/3 of that. Beside the longevity advantages mentioned in previous comments I expect non-attended backup software must be available so I don't have to sit changing cartridges as I now do with Blu-ray disks. (Does anyone know of a consumer-quality Blu-ray writer that takes a stack of blanks and runs unattended?)

I'd appreciate reading more from those respondents who have experience with LTO-n tape drives and media.

LTO Tape (or DLT, or QIC, or 8mm, or 4mm or 8 Track, or Cassette or whatever) is a bit antiquated these days. I'm not saying it's bad to think outside the box or that tape backup wasn't used in a serious way for quite a long time but I think the problems everyone experienced with tape are the reason tape isn't very popular today. In most cases when it came time to perform a recovery, there was much stress and holding-of-breath wondering if a recovery would even work at all. In general, testing was problematic and time consuming. Finding files was time consuming and cumbersome. Same with recovery. Everyone seems to forget that the most important part of a backup strategy involves testing the backup and being confident that the recovery process is not only reliable but that it is also understood how that recovery will take place and test it regularly. This is where tape was the biggest disappointment. In addition, fixing drives when tapes would get jammed or "eaten" and keeping heads clean is also a maintenance headache.

 

[epsiloneri]

LTO Tape (or DLT, or QIC, or 8mm, or 4mm or 8 Track, or Cassette or whatever) is a bit antiquated these days.

On the contrary, LTO is cutting edge in enterprise digital storage, and its usage is increasing. The latest revision from 2012, LTO-6, holds 2.5 TB uncompressed and can be read/written at a bit rate of 160 MB/s - that's faster than single hard drives, and enough to saturate a gigabit network. Future revisions are planned to store increasingly high data densities, up to 48 TB with 1.1 GB/s for LTO-10 (granted some years into the future). Archival reliability for LTO cartridges is far better than hard drives, they are designed for long shelf life. You can easily convince yourself by googling around. Wikipedia gives some good background info on LTO.

A worry for archiving is naturally that there should be drives able to read the media in the future. Given the general (and increasing) usage of LTO and the backward compatibility of newer revisions, I think it is safe to assume that it will be possible to read tapes 30 yr from now without going to expensive extremes. The same goes for optical discs.

I'd appreciate reading more from those respondents who have experience with LTO-n tape drives and media.

I don't have my own tape drive, but at work we use LTO tapes for backups. The only advice I can give is to verify your data once written to tape, and perhaps write two copies to be stored at independent locations. I'd also recommend using WORM tapes for archiving, to avoid mistakenly over-writing data later. Make sure the drive you purchase is compatible with your system. LTO-3 stores 400 GB, and its data rate at 80 MB/s is still much better than you get for BDs. Plus, less swapping of media.