Best Methods For Long Term File Storage ??

[AmbientLight]

RAID is nice enough to protect against disk errors, but it does not protect at all against errors of the RAID controller, which can be damaging to multiple disks in one instant.

I therefore recommend using RAID only for performance optimization and using separate disks/disk storage systems stored ideally at multiple locations.

 

[bkorcel]

I agree. RAID makes sense for certain applications but it's still a single point of failure, flood, fire, theft. You still need to have a copy off site somewhere else. In this case RAID is not necessary and frankly a waste of money. You can save your money and buy a bunch of 3TB USB drives to replicate copies to.

I suppose you could REALLY go out on limb and buy two RAID systems, one to keep offsite at work or a relatives house.

Lots of talk of RAID but I hope everyone makes multiple backups in different locations, I'm pulling this number out of my backside but I suspect RAID protects against under 50% of real-life data loss. That includes viruses, application errors, user error, fires, lightning strikes and theft etc. Can also make recovery harder if you can't replace your EOL RAID controller with something compatible.

Personally I'd only use hardware RAID for performance or where 100% uptime was needed, otherwise spend the extra cash on extra drives for backup and leave a few you only rotate once a year or so in case you need to recover files from an earlier point.

 

[aprotosimaki]

I think it is important to distinguish between data availability and data integrity, since some of this discussion is conflating these two principles. RAID, for example, provides data availability but it does not provide data integrity as does, for example, off site backups.

Think about it this way. If you build a mirrored pair of disks and copy all of your photographs to the raid set, you are still at risk of data loss; it is easy to accidentally delete a photograph and the raid set will not enable you to recover it because the data will be deleted from both disks.

If you want both data availability and data integrity then implement RAID (raid 1, raid 5, etc) *and* then off site backups (off site here defined as media that is not in the same location as your raid set).

In short, you need to back up your data even if you are using RAID. It is a serious error to think that raid provides data integrity.

In my view it would be better to simply not use raid but copy your data daily to different media since data integrity is probably more useful for most people, but of course not all. This is what I do on a regular basis. I have one disk holding my photos and two other disks containing backups of that same data (I am paranoid). If my main disk fails, I am guaranteed of not losing any data and I perform incremental and full backups using these two disks.

 

[etg9]

You could also consider Thecus N7700 series.

It basically seems to be Freenas with a reasonable management GUI in a 7 slot box with support for both Apple and PC and iSCSI et al..

Supports RAID 5 and 6 and now has a 10GE interface.

...

Thecus is another well regarded brand. I believe the box has version control to guard against what people are talking about with accidental deletions, viruses, etc.

The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those.

One of the things I don't like about hard drive pushing is that it's not automagic. You forget to back up to disk after a big download and 36 hours later your back at your computer to find a failed disk that data is gone. My NAS doesn't forget, my computer might bug out and install windows updates in the middle of the day when I'm gone but the operation isn't computer based so backups don't stop. If your backup spans more than one disk you're in more drives to your rotation. I have 7TB of files that I would like to take care of, not all of it goes up to the cloud but I would like to keep it around if I can and that would mean 9 single 3TB hard drives to do active/passive/offsite plus rotating them out so offsite is fresh every 2 weeks or month. too much work for my tastes.

Do you use the 10GBE link? with what switch? I've been thinking about picking up a 24 port dell for home use as it's one of the cheaper ones.

 

[PeterJ]

The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those.

Sure ZFS solves that but it sounds like because of the volume of data you sync over a local network so it's physically close?

 

[etg9]

The snaps of the RAID I was talking about before was exactly this and what I have recommended. They work just like Time Machine or VSS if you are familiar with either of those.

Sure ZFS solves that but it sounds like because of the volume of data you sync over a local network so it's physically close?

Correct, and the out to the Amazon cloud for my offsite.

 

[Ellen Schmidtee]

A few guidelines:

* No backup is forever, so refresh. Hard disks, DVDs, etc have a finite life time. At some stage it would corrupt, so you have no choice except making a fresh copy.

* Keep an off-line copy. An on-line disk exposes the files to viruses, and might be damaged due to power surge.

* Keep an off-site copy. A disk in your apartment might be damaged in fire, earth quake, flood, etc.

I stopped backing up to DVDs, because copying >100GB of data takes a long time, and distributing the work over time (e.g. start disk 1 before going to sleep, start disk 2 before going to work, start disk 3 before going to sleep, ...) makes tracking tedious.

BR would make it a lot easier, but around here both the BR-drives and the media are expensive to the point it's cheaper & faster to use hard disks.

I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.

 

[thexfile]

Backup your files to Blu-ray. 25GB or 50GB discs.

Technology will change before the discs ever go bad.

 

[tron]

I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.

That way you keep fit too ;D

 

[friedmud]

Currently I just use Apple time machine to back up to an external hard drive AND a Flickr Pro account where I upload full resolution versions of all my photos. In the case of catastrophe I could recover my photos from Flickr (albeit not the RAWs)

A good option is to use with Time Machine with two drives. Keep one at a friend's house and swap them out every now and again.

 

[SilverSnake]

I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month. Just ftp all my pictures up there. The only limitation is you can't have more than 250,000 files.

Now this is a really bad idea. Yes, they give you what they call "UNLIMITED webspace", but it's limited to 50GB and gets increased once per day if you near the cap. They offer no guarantee that your data will be available and they perform no form of backup other than their "two geographically separated datacenters".

They also quite clearly say that if they terminate your account, for any reason, they will just toss all your data without giving you a chance to fetch them. Add to that that they expressively forbid you to use their service as a backup service, you are actively breaking the agreement doing so and if / once caught it'll all be gone anyway.

[quote author="1and1 ToS"]3.1.8.

You are responsible for backing up Your Data on your own computer. 1&1 does not warrant or otherwise guarantee that it will back up your data or that data which has been backed up can be retrieved, and will not be responsible for any archiving or backup of Your Data. If any of Your Data is damaged, deleted, lost or corrupted in any way, or becomes otherwise unavailable due to termination or suspension of your account pursuant to this Agreement, 1&1 will have no obligation or liability to you.

8.16.

You shall at all times use Web Site Space exclusively as a conventional Web Site. You shall not use the Web Site Space or Your Services in any way which may result in an excessive load on the 1&1 Equipment, including but not limited to installing or running web proxies, using your allotted space as online backup or storage, or mirroring mass downloads.[/quote]

You are much better off using a proper backup service, there's plenty of them out there and they can be had with unlimited storage for 5US$/month if you pay monthly and almost half that if you pay a year at a go. Don't be clever and exploit web hosting solutions, 1and1 isn't alone in having things like the above in their agreements.

Source for the above ToS quote: http://www.1and1.com/Gtc?__lf=Static&linkOrigin=linux-web-hosting&linkId=ft.nav.tandc

 

[dawgfanjeff]

We don't really have a firm set of requirements here, and obviously that affects the advice.

Archival
I can assume that we want to protect against fire/flood, theft. For these you need offsite backups, or a good safe. Cloud offers automated backup while you're sleeping, but slow and not for free, on the upside, they handle all the messy RAID stuff for you. CD/DVD/BD sound great but are extremely time consuming and tedious, and you still have to be disciplined to physically take them someplace.
Local NAS. On local network, so GB backups are fast. You can run them on demand, and/or scheduled. AND you don't have to worry about paying for bandwidth used. Downside, not as turnkey as some would like. I have a netgear readynas ultra 2. Two disks, mirrored (RAID 1). Best for redundancy, not for performance. Like other NAS solutions, it supports lots of other features, has an ftp server, DLNA, etc.., etc...lots more. Not saying its the best one in the world, but it's what I went with. I use robocopy scripts to copy data to it.

Note, for scripts, onsite backups are fine, but it needs to be additive, NOT with a sync, else you run the risk of replicating the accidental deletion (d'oh!) or simply overwriting non corrupt data with corrupt data. If you are using robocopy, that means you want /e /s but NOT /mir.

Retrieval/Media
How fast do you need to get the data back? An hour? Same day? A month?
For local workstation crash, accidental deletion. NAS gets the data back, fast and free, but of course there is upfront cost. Cloud gets the data back, ondemand, but SLOW, and might be subject to data consumption charges. CD/DVD/BD...gets the data back, but slow b/c you have to go get it. Tape I discount entirely. Too tedious, expensive and quite subject to obsolescence. If you'd archived with top of the line SCSI tape system 10 yrs ago, you'd be kicking yourself today trying to use it to restore data.
I don't have a solution for this one either...yrs ago a few of us talked bout hosting ftp sites for one another at each other's houses, but that is hardly permanent/reliable either, plus the bandwidth hogging issue. I am thinking about 7zip ing all of it (with verify switch) and dumping it to USB3 removable storage and putting it in a safe at home every so often. Of course, neither USB nor sata is future proof, but its the closest thing we have right now, and it's cheap/scriptable.

One note about defragging...it's on by default in Win7, and presumably, Win8. Not sure diskkeeper is adding any value. In Windows server, you can schedule it with defrag.exe.

 

[etg9]

... I have a netgear readynas ultra 2...

ReadyNAS was developed by a company called Infrant, they were aquired by Netgear. Up until they were aquired they were considered one of the best quality NAS boxes you could get. They don't have the sweetheart status they used to but they are still a great product.

I moved from Infrant to Synology and have been extremely happy.

I also agree with most of what you said, it's hard to tell someone what the best thing to do is when we don't know what he is trying to accomplish.

 

[And-Rew]

Taking the OP at its basic question - i work with 5 EHD's as follows:

I work with an iMAc so have a large EHD plugged into permenantly to be used for Time Machine back up - i really love this back up system.

All image processing is done via Adobe LR so, files are imported via LR with an untouched copy being backed up to a second EHD. This EHD is backed up to a 3rd EHD via Chronosynch.

So, that's the original files covered.

All processed images are held within the LR database with side files on the iMac, but once an image is ready to be used - a copy is exported to a 4th EHD which, you guessed it, is backed up to a 5th EHD via Chronosync.

EHD's 2 and 3 are stored 1 at home and 1 at a family members with the same applying for EHD's 4 & 5.

Cost of this solution is a whopping £400 for all 5 EHD's and a copy of Chronosync. Time Machine comes free with Mac OS X and all hard drives are of the 1Tb or 2Tb capacity. As i run out of capacity, i epxect i will upgrade to a pair of 4 or 5Tb EHD's (by the time i run out of space) for another £150 in a few years time. Yep - still doing it cheaply

So, the real price of all this is some cheap (and always getting cheaper) EHD's and a bit of time on my part - but that time can not compete with the cost of losing all any of my files :'(

 

[Mt Spokane Photography]

@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.

I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years. Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

 

[dawgfanjeff]

I don't love the printing idea; however...all those pictures will be the most valuable thing you have when the zom-pocalypse occurs! Except bullets of course ;D

 

[dirtcastle]

@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.

I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years. Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

Agreed!

 

[etg9]

@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.

I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years. Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

Agreed!

We just really have no idea, 100 years ago they probably weren't sure the prints would be around now as the first prints in the mid 1800's didn't hold up so well. RAW is a little funkier as it has to get processed when opened but JPEG is a much more developed and solid standard. I can imagine that will be around in 100 years as it's already been around for 20 and it doesn't seem like it's going anywhere.

 

[thedge]

Amazon glacier starts to get expensive as you add more data. 500GB of storage seems to work out to $5 USD a month. I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month. Just ftp all my pictures up there. The only limitation is you can't have more than 250,000 files.

FYI, if you read the terms of service for 1&1 (and most or all webhosts), using the FTP space as a backup is not allowed. Them finding out is different, but it is against the ToS.

 

[mws]

Not by any means a feasible solution, but I wonder how many sheets of paper I would need to print out all my photos in a series on 1s and 0s………...