Best Methods For Long Term File Storage ??

[Halfrack]

http://aws.amazon.com/glacier/

Glacier is an amazing service, literally, in the cost of backing up huge amounts of data 'just in case'.  The catch is all in the amount of time it takes to upload all of what you currently have, depending on your internet connection.  It'll cost you a fair amount to get the data back, but compared to 'never' it's damn cheap.

The dual drive solution is an easy step 1 for most folks.  If you're looking for a cool tidbit, Sentry Safe makes a few models that have a USB2 pass thru - so the drive can be in the safe and yet online at the same time.  The key to these 'data rated' safes is that the internal temp is much lower than other safes, and it's rated to how long it will keep the insides under 120 degrees.

Tape is a funny medium.  It's amazing in that you don't have to worry about motors and stuff like with hard drives, but the drive itself is a huge expense.  It is the only method to look at that has 20+ year archival bits - that and MO or UDO drives and media (small, expensive).

It comes down to why are you saving the images. 

Kids/family/vacations?  Dual hard drives, maybe toss some onto USB flash memory cards (or old 1/2/4gb SD/CF cards - great way to justify faster/bigger/new cards).  Lots of these are online as part of your smugmug/flickr/picasa/etc gallery right?

Paying work for clients?  This is where you need to consider multiple mediums and methods.  Online may not be acceptable, so you will want something that's plugged in and running (a NAS with redundancy (RAID) and a good filesystem (ZFS)), plus a copy that's not spinning.  Depending on the price point charged, having it archived to tape may be a safe cya, or encrypt it and send it up to glacier.

[neuroanatomist]

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Makes sense, thanks!

[Dylan777]

Always keep your files duplicated.  Any kind of media is fine.  However, you have to ensure the files you backed up is always good.  DVD is not my suggest.  Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files.  Do not waste your money to buy too big external drive because hard drive price drops very fast.
Here is my setting.
* 6 2TB hard drives @ RAID 5 for master drive
* 1 3TB external drive for backing up important files
* Using MS SynToy to keep my files always updated.

+1....I used to pickup 2-3TB drives, but I now I going for min 5TB. It's getting cheaper everyday though. Keep one at home and other at my in law house with monthly backup.

[bkorcel]

More important is not the disk longevity but after 5 years when your DV burner stops working, you will buy a new one and there is a very high probability that it will not ready your DVD's.  I've had instances where it can read sectors close to the spindle but as you move out, it gets out of spec.

I do not rely on CD or DVD media anymore.  I keep my RAW and processed files duplicated on external USB 3TB drives and my processed files go out on the cloud.  All originals get copied to at least two locations before I format the cards in the camera.

I've had an instance where a USB drive failed....because they do!  Fortunately under warranty and I was able to robocopy the original onto the new drive.

Also note that magnetic media does not last forever either.  Every few years you will need to dump the contents onto another disk, reformat, dump them back.

The cloud is my final last resource to prevent loss of processed images.

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

[Halfrack]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

[neuroanatomist]

...I know it's not forever.

But what is?  I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy?  ...dating myself again, I know).  The data may be intact, but I have no way to read them...

[ahab1372]

...I know it's not forever.

But what is?  I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy?  ...dating myself again, I know).  The data may be intact, but I have no way to read them...

Common problem, and a huge one for archives, libraries etc.
Every time we upgrade or dispose of hardware and/or software, we should be checking if there is data somewhere that we should be converting or copying. I'm not obliged to archive everything, and most files that I haven't looked at in a year or two, or was created with software that I cannot run anymore, are probably not worth keeping. But for pictures I might judge differently.

[cliffwang]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

[Drizzt321]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?

[cliffwang]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?

I used to have FreeNAS as my NAS.  However, my wife complains I had too many systems.  Thus, I have to reduce my toys.  I have already phased out two systems.  We still have few systems at home.  My target is to have only one desktop(server) and two Windows 8 PRO tablets.  If I build another server, I will get trouble with my wife.

[Drizzt321]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?

I used to have FreeNAS as my NAS.  However, my wife complains I had too many systems.  Thus, I have to reduce my toys.  I have already phased out two systems.  We still have few systems at home.  My target is to have only one desktop(server) and two Windows 8 PRO tablets.  If I build another server, I will get trouble with my wife.

Hmm...maybe have one big honkin' machine that you run virtualized Windows on for your Windows needs? I've recently had the thought that maybe I'll do that for my Lightroom/PS needs...still haven't made up my mind on investigating that yet, but it's crossed my mind.

[Promature]

I recommend Synology and WD Red Line drives.  Synology is a great "NAS" and if you get a 2 bay solution, you can add a second drive later.  Synology software is also much better thatn Windows Home Server 2008.
http://www.amazon.com/gp/product/B008JJLZ7G/ref=ox_sc_sfl_title_5?ie=UTF8&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B005YW7OLM/ref=ox_sc_sfl_title_6?ie=UTF8&smid=ATVPDKIKX0DER

[kdw75]

Because of changing media and technology I tend to favor cloud storage. I keep a copy on my computer and when my drive fills up I put in a larger one and put the old one in a drawer. This way I can always revert back to older drives if necessary. Then I also keep my stuff stored with a backup service since keeping it in that form doesn't rely on me having a particular computer or medium to read it. It is accessible any where I go with an internet connection. I probably should backup off site, but I haven't gone that far. I plan to get a RAID system at some point.

[gmrza]

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Makes sense, thanks!

A bit of reading I did into DVD/BluRay media recently basically indicated that there are no real standards for archival optical media, and quite honestly, your mileage may vary. - It is rather uncharted territory, and there seems to be no consensus as to how long a burned DVD or Bluray disk could last.  (I have some CD-Rs which are around 20 years old, and still readable. - Is that pure luck?  I have no idea.)

The bottom line is that for data backups you need to have a policy of copying to new media every few years.  That however tends to happen by default as existing drives become too small and larger, cheaper drives hit the market.

To be heretical: the most reliable long-term storage medium for a photo still seems to be paper. Provided a print is stored under controlled conditions, you may achieve the rated life of around 200 years for archival paper. - I don't think any of us would be around to confirm.  Of course, that does not answer the OP's question!

[CharlieB]

I dump to primary HD on desktop.  It automatically copies to secondary on my Windows server.  The Windows server cross-pollinates with Linux server nightly.   Done.  Three copies, three machines, three hard drives.

CD... dont last, unless you do a slow burn on them.  I've got two and three year old CD's... lots of problems with those.