Best Methods For Long Term File Storage ??

Status
Not open for further replies.
cayenne said:
cliffwang said:
Halfrack said:
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad! ZFS does not apply on Windows system. That's the file system I really want for Windows environment. ReFS is the only solution for Windows system. I will try Windows Server 2012 next month after I return from my vacation.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

cayenne

ZFS applies to Windows very well - it's over the network, same as Windows Server 2012 (you're not going to use 2012 as your desktop OS are you?). WinFS was supposed to be the end all, but it got dropped.

Too bad there are minimal if any options in the small form factor boxes that hold 4-6 drives. If you have been doing giant servers, your wife is correct in making you downsize - it's for your own good :D
 
Upvote 0
NAS (Raid 5 or 6) to Cloud is what I safe is the safest bet. That covers corruption, accidental file damage, disk failure, enclosure failure, catastrophic damage, and regional damage. International damage would not be covered but if the US is gone then I don't need my pictures anymore.

I have a Synology NAS at home and love it. It's small and quiet and has lots of options. I have it keep 5 snaps of any changes so I can revert if need be. There are cheaper things to buy, but their are tradeoffs and this one fit my needs. Pick a provider with a good long term track record and replace the NAS every 4-5 years

After that data gets sent off to the cloud for an offsite backup. I use Amazon Glacier now but have jumped around and another service may fit your needs better. I copy compressed JPEG only to Skydrive as a complete last ditch about every quarter.

My photo backup is currently about 500GB (I don't keep too many raw) so that's only $5/mo. To get it all back out of there would be about $60, for DR this is cheap and Amazon is a good trusted name.
 
Upvote 0
Lots of questionable advice, here.

First, do not rely on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever does own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.

If you're not overly worried about your cloud provider accidentally or intentionally sharing your data with the world, it can make a nice additional supplement to your data archiving strategy, but only as a "if everything else I'm actually relying upon goes tits-up, I'd hopefully still be able to get to it in the cloud" sort of last gasp hope.

With that out of the way, the only reliable method is to continually keep all your data readily available and online and part of your regular backup routine. As others have pointed out, old media die in lots of different ways. If you copy all your old media to whatever you're using today, you don't care if the old stuff dies for whatever reason, and you're also confident that you've got a valid copy. There's no worry that your several-year-old DVDs might be starting to delaminate, or that your Zip drives will have the click-o-death, or whatever.

Yes, that means you need bigger hard drives today, but the good news is that hard drives are dirt cheap compared to whatever you spent on your old media. A single DVD doesn't even store as much as a typical CF card. A hundred bucks gets you a hard drive that holds the equivalent of a few hundred DVDs. When that drive fills up, get another.

The simplest and most reliable backup method these days is to get three times as much disk space as you need. Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.

Also worth investigating, depending on your performance needs and your desire for tidyness, are RAID arrays. Be careful; many commonly-used RAID modes actually put you more at risk for data loss than a single hard drive, meaning you need that much more redundancy in your backups to compensate. Safer RAID modes eat up more disk space. Duh! But you only want to think about RAID if a single disk isn't big and / or fast enough to hold all your stuff, and you should then think of the RAID array as a single disk that happens to have some extra moving parts.

IOSafe also makes near-indestructible hard drives: fireproof, waterproof, crushproof. They're more expensive than a regular hard drive, but very reasonably priced. If you're on a Mac, just get one (or three or however many you need) and point TimeMachine to it (them), and the only reason you'd need an offsite backup is if you're worried about theft.

Cheers,

b&
 
Upvote 0
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

When I hear stories of people backing up to DVD, the first thing I always think is, oh my god, how much free time do you have? I have 140 gigs of pictures and I'm sure that's a lot less than a true pro. Backing that up to DVD would require 28 disks for crying out loud.

Drobo makes a pretty clever solution for all this.


As to the suggestions for FreeNAS...I haven't used it in two or three years, but when I did, I thout it was a debacle. It was far too complicated and required far too much user configuration. I think this was back in the version 7 days...has it improved since then?
 
Upvote 0
Mar 25, 2011
16,847
1,835

While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive.

\
You are aware that some lost everything in the recent Amazon crash?? Some lost their files when their online photo services folded. Online storage has not really been proven to be a reliable system that you can upload to and expect to have your data 50 years from now. Nor is a hard drive, raid system, SSD, memory card, DVD, or CD.
You must actively keep multiple backups and move data as media becomes obsolete. That was my point, nothing is as reliable as those 100 year old prints that many of my cousins have in a shoebox in their closet. Some may be lost, some may burn up, but there seem to always be someone who managed to save them. I guess thats a case of many many backup copies to a media that does not lost the images after 50 or 100 years.
http://www.businessinsider.com/amazon-lost-data-2011-4

My Grandfather from around 1905. You had to sit very still then, no fast lenses or shutters.

1169914314_vevDC-M.jpg


My Grandmother born in 1886
1169911783_awWaN-M.jpg


The point is that we still do not really have media that can match the longevity of the old prints from 100 years or more ago, except to make and store prints.
 
Upvote 0
You do have a point but that outage was with EC2 not glacier. I'm not sure what the promised durability for EC2 is but glacier is very high due to the fact that it's stored in multiple facilities and multiple hard drives within that facility. I have not had faith to offer retaining data to my clients but Amazon's Glacier looks promising and relatively affordable. I'll of course keep a local hard drive with the data on it. But as you say, that's not enough. With Glacier, I can feel secure that even if my house burns down I'm ok. And I don't have to worry about trying to put data in other locations other than on Glacier.
 
Upvote 0
@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.
 
Upvote 0
J

JerryB

Guest
It might be the only effective way to save our photos is to print them out. Hard drives fail, DVD's, and CD's only last so long, and who knows if any of these will exist in a few years. But prints will still be there.
Many companies use RAID array's which consist of multiple drives to backup their files. Some like, credit card companies, have a backup of their array's located halfway around the world.
 
Upvote 0

tron

CR Pro
Nov 8, 2011
5,222
1,616
Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
First person answers: "I also have a monthly card" ;D

Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...
 
Upvote 0
B

bkorcel

Guest
Now that you mention it, Many safes now are coming with integrated power and USB hubs so you can lock up your hard drives or small servers inside to protect from theft. Some are even fire rated.

tron said:
Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
First person answers: "I also have a monthly card" ;D

Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...
 
Upvote 0

cayenne

CR Pro
Mar 28, 2012
2,866
795
Leadfingers said:
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

When I hear stories of people backing up to DVD, the first thing I always think is, oh my god, how much free time do you have? I have 140 gigs of pictures and I'm sure that's a lot less than a true pro. Backing that up to DVD would require 28 disks for crying out loud.

Drobo makes a pretty clever solution for all this.


As to the suggestions for FreeNAS...I haven't used it in two or three years, but when I did, I thout it was a debacle. It was far too complicated and required far too much user configuration. I think this was back in the version 7 days...has it improved since then?

freeNAS looks pretty straightforward from what I'm seeing.

I'll try to report back when I get the box constructed and running....

It is version 8.x.x something now.....

C
 
Upvote 0
As an IT guy who designs DR (just a part of what I do) for a living and like I said earlier in this thread a working copy on site, a back up on site and a backup offsite and in many locations as possible is the safest bet.

I never told anyone to use the cloud as a single backup method. Tape is a poor backup solution for long term, as are DVD's but for different reasons. DVD's tend to get scratched or wear out, tapes no one has that model tape drive 10 years later and they are not backwards compatible. Single external hard drives die all the time.

Keep you data yourself, on a NAS. Use the cloud as a backup to that. Very easy to manage and very hard to kill.

Print is a bad idea for archival, I'm happy MSP's photos are around. a lot of people's aren't: fires, floods, tornadoes, earthquakes, burglars, or just getting lost have claimed lots of photos. They also take lots of room and are hard to move about and organize. This may work as a last ditch backup of very important family photos.

People telling you to keep backups in bank boxes or at parents houses tend to live close by, no geographic dispersion. that big earthquake/flood/fire may very well hit them too.


TrumpetPower! said:
do not rely on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever does own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.

Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.

But you only want to think about RAID if a single disk isn't big and / or fast enough to hold all your stuff, and you should then think of the RAID array as a single disk that happens to have some extra moving parts.

...and the only reason you'd need an offsite backup is if you're worried about theft.

The Cloud, read the TOS for the provider and make sure it fits your needs. don't make the cloud (or anything) your single backup source.

The disk swap is a ton of work, hopefully the last backups were good and hopefully the disk didn't fail. when you fill up one disk you get to buy 3 more. don't do this.

RAID (other than 0-1) also protects against hardware failure with an extra hot spare (5/6) or array (10/50). A NAS also doesn't have to be a single drive, the space can be logically separated.

or any other natural disaster.
 
Upvote 0
I currently use RAID 1 (mirroring) on 2x2TB Drives in my primary computer just for image storage. I have SSD's for my OS and primary Apps and non-redundant HDD's for non-critical files.

I keep an active 3 TB External connected for nightly scripted backup of the RAID 1 files.

I keep another 3 TB External at work that I bring home once a month to backup and store as my off-site.

I replace all my drives every 2-3 years to reduce the risk of aging drive failures.

In the event I have any failure in either drive in the RAID 1 array, the other is still fully functional. I perform an immediate backup to the external, grab a spare 2TB spare, and swap the problem drive out and allow it to automatically rebuild the mirror image on the new drive.

Perhaps at some point I may consider cloud but have found that my current system has served me well and survived a hard drive failure once already. If I am able to increase client level work, then I will probably make the move to cloud storage as my off-site method.
 
Upvote 0
Jan 21, 2011
523
1
Leadfingers said:
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium. On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

One point to take into account especially with RAID5 or RAID6 - you probably want to make use of enterprise drives (not desktop drives) to be sure that you get the reliability you want.
 
Upvote 0
You could also consider Thecus N7700 series.

Refer to http://www.thecus.com/product.php?PROD_ID=14

It basically seems to be Freenas with a reasonable management GUI in a 7 slot box with support for both Apple and PC and iSCSI et al..

Supports RAID 5 and 6 and now has a 10GE interface.

I run a 7x2TB and a 7x3 TB setup which yields 8TB and 12 TB formatted data space. I prefer RAID 6 as it can take more than 12 hours to rebuild a failed array. This puts a RAID 5 setup at risk during the rebuild window.

The Thecus does only have 1 power supply, so there is no redundancy and hence a single point of failure, but it has been good enough for SOHO use. In three years I have not had any failures except for drives.

I prefer normal Seagate drives, not the more expensive Enterprise quality, they last between 14000 and 18000 hours, that is between 1.5 and 2 years, and the price is OK. In my setup WD has had recurring failures in the first 1000 hours.

Swopping drives are easy. The system is hot pluggable so you simply unplug the drive carrier, loosen the four screws that hold the failed drive, replace drive, refasten and plug back. The system detects the new drive and functions in Degraded mode until fully rebuilt. No real impact on user experience.
 
Upvote 0
Lots of talk of RAID but I hope everyone makes multiple backups in different locations, I'm pulling this number out of my backside but I suspect RAID protects against under 50% of real-life data loss. That includes viruses, application errors, user error, fires, lightning strikes and theft etc. Can also make recovery harder if you can't replace your EOL RAID controller with something compatible.

Personally I'd only use hardware RAID for performance or where 100% uptime was needed, otherwise spend the extra cash on extra drives for backup and leave a few you only rotate once a year or so in case you need to recover files from an earlier point.
 
Upvote 0
Status
Not open for further replies.