Best Methods For Long Term File Storage ??

[1255]

Hi all.

Would like to ask for your thoughts on best methods for long term file storage, meaning, external hard drives or DVD/Blue Ray? Or something else?

I had always thought that external hard drives were best, at the very least because who knows when we'd need to convert those DVD or Blue Rays to a new technology.

Then I just read in another post that external hard drives corrupt files over time. Is this true? I had always thought that it was the process of dragging files and/or copy and pasting them over and over that impacted data bits and degraded the files.

What do you think is the best method for long term file storage?

As always, thanks to all of you for your input.

Much appreciated.

 

[Drizzt321]

Hard drives suffer from bit-rot (as do some other forms of storage). It can occur in a couple of cases, such as improper storage, flooding, etc, or cosmic rays, or other similar high energy events.

There are a couple of ways to mitigate this. First, keep a copy on several different storage devices, preferably of different types. This helps mitigate one of them going bad. If it's all hard drives, get ones with serial numbers which aren't right near each other, which can help mitigate if a particular lot of hard drives has issues (it happens, manufacturing isn't perfect).

Further, you can use a filesystem or storage mechanisms which provide a high level of error correction and detection. You can use something like Par2/Parchive which provides parity and checksuming which can detect, and often correct for bit-rot errors.

A good archiving solution will including multiple storage media, and proper storage of that media. In general, a high quality tape backup with proper storage tends to be the best, very long term archival mechanism and is still very popular and widely used in various enterprise backup/archive solutions.

 

[1255]

thanks very much

 

[Standard]

I simply archive all my files onto DVDs. If there are important files that I worry about, I make sure to archive them onto several copies for redundancy (I do this over a period of time) so as to avoid possible disc errors. All discs are then catalogued (with a cataloguing app for quick search and retrieval at a later time), then stored onto large 200+ disc binders and placed in metal file drawers. Simple as that. No issues thus far. And yes. I archive an extra copy of select important files onto Lacie 1+ TB external hard drives.

 

[Iahcon]

I keep two copies on separate external hard drives.

 

[neuroanatomist]

I keep two copies on separate external hard drives.

As do I, and the two HDDs are stored in different physical locations (home and work). Those copies are made as soon as the images are processed (and until then, they're still on the CF cards - since getting the 1D X, I always have at least 2 copies of every image). Quarterly, I make a DVD backup that goes in a safe deposit box in a third location (different town than home or work).

 

[florianbieler.de]

Got mine synced to a 2,5" external hdd which i use not that often.

 

[Crapking]

What about on-line "cloud" storage sites which allow access without physically requiring the disk drive?
Of course this is in addition to the physical backups, but I use Phanfare.com {with a reasonable annual fee}, which allows web-based access, including RAW storage (for an extra fee) for the really important files.

 

[myocyte]

I've been meaning to sign up for this, but Amazon provides a great, affordable solution for cloud storage. If anyone has tried it, I'd be interested to hear about their experience.

http://aws.amazon.com/glacier/

 

[sb]

I keep two copies on separate external hard drives.

As do I, and the two HDDs are stored in different physical locations (home and work). Those copies are made as soon as the images are processed...

Same here with a small difference - I make a copy immediately after dumping the files on the primary computer. Processing is all in LR metadata anyway, and LR catalogues are also included in my backup script.

My backup script is a hand made batch file using the "Robocopy" command line utility.

My back up drives reside on a second computer which is identical with all the same applications installed as the primary. That way if my primary fails, I can continue working uninterrupted.

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years. I also don't use external drives because for the price of a "network storage" I can build myself a fully functional second computer, so what's the point?

 

[neuroanatomist]

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'? I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.

 

[Drizzt321]

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'? I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

 

[Emeyerphoto]

1. Dump on to external as soon as get home.
2. Modified images get saved on secondary external.
3. Backblaze backs them up onto the cloud, can recover them anywhere, and if necessary I can receive an external hard drive with my files on them for a fee.
4. Relax, since they are on the cloud and do not need to mess with safety deposit boxes, nuclear proof storage, or 10 external hard drives of the same data.

 

[cayenne]

Lots of good posts here.
ONE thing that should stand out...redundancy, and multiple backup places and methods.

CDs/DVDs/BRs...all can degrade, and depend also on the quality of your starting materials...cheap will usually rot quickly.

I'm looking to build a fairly large, redundant NAS (Network Accessible Storage) system. I'm looking to use the freeNAS system:

http://www.freenas.org/features/feature-overview/category/features

Wikipedia:

http://en.wikipedia.org/wiki/FreeNAS

What makes this nice...is using the ZFS filesystem, which allows you to add different sized / speed drives over time...and can be hot swappable....and can lose 2 (or more depending on set up I believe) drives at the same time in the system, and still not lose data.

I just bought a special at newegg the other day, a barebones computer I'm about to assemble for about $168...8GB ram (which I'll likely upgrade to 16 for about $50 or so)...

Anyway, set that up with a 5 disk enclosure I bought awhile back to start with, and build it from there. In the future to have the comfort of redundancy offiste , in case of fire, flood, hurricane (I live in NOLA after all), I'm going to set up a similar system at my mom's house out of state...and that way, I can justify paying for her internet connectivity since it will be a 'business expense'.

Anyway, I'm getting off topic...but if you don't mind a little DIY spirit....take a look at the links and systems I've listed above. It is very high quality stuff, and can be turned into a highly reliable, flexible system that most any computer you own can connect to and backup, whether it be a Windows PC, or Linux box...or even OSX on a mac with time machine, which can often be finicky at times on its own right.

HTH,

cayenne

 

[cliffwang]

Always keep your files duplicated. Any kind of media is fine. However, you have to ensure the files you backed up is always good. DVD is not my suggest. Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files. Do not waste your money to buy too big external drive because hard drive price drops very fast.
Here is my setting.
* 6 2TB hard drives @ RAID 5 for master drive
* 1 3TB external drive for backing up important files
* Using MS SynToy to keep my files always updated.

 

[Halfrack]

http://aws.amazon.com/glacier/

Glacier is an amazing service, literally, in the cost of backing up huge amounts of data 'just in case'. The catch is all in the amount of time it takes to upload all of what you currently have, depending on your internet connection. It'll cost you a fair amount to get the data back, but compared to 'never' it's damn cheap.

The dual drive solution is an easy step 1 for most folks. If you're looking for a cool tidbit, Sentry Safe makes a few models that have a USB2 pass thru - so the drive can be in the safe and yet online at the same time. The key to these 'data rated' safes is that the internal temp is much lower than other safes, and it's rated to how long it will keep the insides under 120 degrees.

Tape is a funny medium. It's amazing in that you don't have to worry about motors and stuff like with hard drives, but the drive itself is a huge expense. It is the only method to look at that has 20+ year archival bits - that and MO or UDO drives and media (small, expensive).

It comes down to why are you saving the images.

Kids/family/vacations? Dual hard drives, maybe toss some onto USB flash memory cards (or old 1/2/4gb SD/CF cards - great way to justify faster/bigger/new cards). Lots of these are online as part of your smugmug/flickr/picasa/etc gallery right?

Paying work for clients? This is where you need to consider multiple mediums and methods. Online may not be acceptable, so you will want something that's plugged in and running (a NAS with redundancy (RAID) and a good filesystem (ZFS)), plus a copy that's not spinning. Depending on the price point charged, having it archived to tape may be a safe cya, or encrypt it and send it up to glacier.

 

[neuroanatomist]

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'? I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Makes sense, thanks!

 

[Dylan777]

Always keep your files duplicated. Any kind of media is fine. However, you have to ensure the files you backed up is always good. DVD is not my suggest. Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files. Do not waste your money to buy too big external drive because hard drive price drops very fast.
Here is my setting.
* 6 2TB hard drives @ RAID 5 for master drive
* 1 3TB external drive for backing up important files
* Using MS SynToy to keep my files always updated.

+1....I used to pickup 2-3TB drives, but I now I going for min 5TB. It's getting cheaper everyday though. Keep one at home and other at my in law house with monthly backup.

 

[bkorcel]

More important is not the disk longevity but after 5 years when your DV burner stops working, you will buy a new one and there is a very high probability that it will not ready your DVD's. I've had instances where it can read sectors close to the spindle but as you move out, it gets out of spec.

I do not rely on CD or DVD media anymore. I keep my RAW and processed files duplicated on external USB 3TB drives and my processed files go out on the cloud. All originals get copied to at least two locations before I format the cards in the camera.

I've had an instance where a USB drive failed....because they do! Fortunately under warranty and I was able to robocopy the original onto the new drive.

Also note that magnetic media does not last forever either. Every few years you will need to dump the contents onto another disk, reformat, dump them back.

The cloud is my final last resource to prevent loss of processed images.

I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'? I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

 

[Halfrack]

Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written. A lot of times you can't tell if a file is corrupted until you attempt to open it. This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4). ZFS does a good job with this, but would only be used on a NAS type of setup.