October 22, 2014, 11:53:05 PM

Author Topic: Best Methods For Long Term File Storage ??  (Read 17375 times)

dirtcastle

  • Canon 6D
  • *****
  • Posts: 391
    • View Profile
    • Eric Nord Flickr Page
Re: Best Methods For Long Term File Storage ??
« Reply #30 on: November 13, 2012, 10:23:58 PM »
If money is not an issue, I would go with a "cloud" solution. The main two problems with cloud storage is moving the files and host reliability. The issue of upload/download speed can be solved with a Fios or a fast cable connection. The reliability issue can be solved by having two cloud accounts and also have a local backup. Or you could do the double backup on your end and use a single cloud host.

While it's true that there will be some "bit rot" on large collections being stored over decades of time... unless you are the Smithsonian, I don't see why a little loss would be a major problem.

I think the main problem with long term storage is organization. By using a reliable, long term, cloud host, you can have a single storage solution with easy access and no barriers to organization. All existing "local" storage methods will have limits of both size and ability to organize.

If money is an issue... forget long term storage: carefully pick your favorites and buy cheap arrays. Reliable long term storage for large collections costs money; there's no way around it (at least, not that I know of).
« Last Edit: November 13, 2012, 10:26:38 PM by dirtcastle »

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #30 on: November 13, 2012, 10:23:58 PM »

Hillsilly

  • 1D Mark IV
  • ******
  • Posts: 781
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #31 on: November 13, 2012, 10:30:03 PM »
I'm hedging my bets.  I print my favourite photos at 5x7 or 8x12.  I upload photos to an online site.  I keep backups on multiple hard drives.  I also have photos stored on the hard drives of a couple of computers.  With film, I keep negatives in archival plastic sleeves. 

But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish.  Is anyone ever going to look at that many photos or care for such a collection?
1000FN | 7E | 3000 | 3 | LS-100TS

papa-razzi

  • Rebel T5i
  • ****
  • Posts: 146
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #32 on: November 13, 2012, 11:17:34 PM »
But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish.  Is anyone ever going to look at that many photos or care for such a collection?

There will be data mining software that will recognize faces and other image content to pull up specific photos of people, events, etc.
6D | 7D  | EF-S 17-55mm f/2.8 IS USM | EF 24-105mm f/4L IS USM | EF 70-200 f/2.8L IS USM | EF 70-300 f/4-5.6L IS USM |
EF 35mm f/2 | EF 50mm f/1.4 | EF 85mm f/1.8

cayenne

  • 1D X
  • *******
  • Posts: 1212
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #33 on: November 14, 2012, 10:43:42 AM »
But what really worries me is that, at the rate I'm shooting, I'll have hundreds of thousands (maybe a million+?) images by the time I finish.  Is anyone ever going to look at that many photos or care for such a collection?

There will be data mining software that will recognize faces and other image content to pull up specific photos of people, events, etc.
You might wanna all read the TOS for the Cloud service you are using......THEY just might be using said data mining software using your images already for their own purposes.

I'm fairly sure three letter agencies are likely doing it quite prevalently...too large of a free data store for them to pass by.

Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • *********
  • Posts: 8870
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #34 on: November 14, 2012, 11:19:38 AM »
What do you mean by "Long Term"  Do you mean hands off as compared to periodic maintenance and renewal?
 
5yr?, 10yr?, 50 Yr?, forever?
 
There really are no 50 or 100 year hands off storage solution, too many issues can and will happen over that time, and you will likely have to move image files to different media every 10 years or so.
Solid State Memory - forget it
Hard Drives - better, but no
Online storage - no
Prints - possibly
 
Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all.  In about 2000, my hard drive crashed and everythinng was lost.  I had a one year old backup on CD of my images, and recent backups on floppy disks.
 
The CD's had deteriorated in only one year and were unrecoverable, but I did recover most from the floppy drives.  I then started making copies to hard drives as well as CD's and DVD's. I also switched to a raid system.  Since then, I've never had a crash that lost data, but its a matter of when, not if.
 
I have a box full of 100 year old B&W prints and another box of negatives going back to 1940 of my ancestors.  The color prints starting in the 1950's are faded away, and I have some color negatives from the 1940's that have faded as well.  Some of the prints have been copied and sent to me by relatives, so the process of distributing images to relatives who might want to keep them seemed to work. I have scanned them and backed up the scans as well.
 
Current printing technology has been said to produce 100+ year life prints, so that might be the best backup method, prints and a fireproof box.  Send copies to relatives as well.  Many may still be arounnd in 100 years when the files have long been lost.
 

revup67

  • 5D Mark III
  • ******
  • Posts: 627
  • Memories in the Making
    • View Profile
    • Revup67's Flickr Images
Re: Best Methods For Long Term File Storage ??
« Reply #35 on: November 14, 2012, 12:40:55 PM »
With respect to all of these ideas I wish to comment in that how are you backing up?  Is it a straight copy without verification?  Possibly a bad move.  After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50.  There are other tools in the suite as well for partitioning, etc.  You should be doing a data verification at the very least if using a copy method.  Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons.  Paragon will actually tell you if a drive is faulty as it checks the drive before backup.  I do IT for a living and this actually happened at a client's.  Ultimately I replaced her drive.  In addition, I use Diskeeper.  When files are created fragmentation takes place.  In NTFS you have 4096 bytes in each cluster size.  Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order.  When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive.  Solid state drives are not excluded from fragmentation of course.  Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more).  It runs real time, small footprint and no performance impact.  All HD drives on a single PC are kept at 0% fragmentation.  You can run all kinds of reports on the drive.  There are various indicators that will tell you the drives health as well.  Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance.  I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's.  I've got a link to disk keeper on my site if anyone is interested drop me a note.  I can get you a demo. PS they are now called Condusiv Technologies.  Sorry if this sounded like an ad but its my personal experience and using these products daily.  Been using Diskeeper for approx. 9 years and know it well.  It's on every server I manage.
Thanks
Rev
Canon EOS 5D MK3 + EOS 7D » Canon50mm 1.4 » Canon 100mm IS USM L Macro » Canon 24-105 L USM IS » Canon 70-300 IS USM L» Canon MP-E65 » Canon 8-15mm Fisheye »Canon 16-35 Mkii USM L » Canon 200mm 2.8 L USM II » Canon 400mm 5.6 L » Canon 580EXII & MT 24EX Flash

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Best Methods For Long Term File Storage ??
« Reply #36 on: November 14, 2012, 01:32:12 PM »
With respect to all of these ideas I wish to comment in that how are you backing up?  Is it a straight copy without verification?  Possibly a bad move.  After using several different backup programs I found that Paragon's Hard Disk Manager Suite was an excellent product based on the what the product offers, their client base and tenure in the business and cost is a mere $50.  There are other tools in the suite as well for partitioning, etc.  You should be doing a data verification at the very least if using a copy method.  Without verification you've got no way of knowing whether the file(s) were copied properly for a variety of reasons.  Paragon will actually tell you if a drive is faulty as it checks the drive before backup.  I do IT for a living and this actually happened at a client's.  Ultimately I replaced her drive.  In addition, I use Diskeeper.  When files are created fragmentation takes place.  In NTFS you have 4096 bytes in each cluster size.  Each RAW or JPG or TIF (etc) can have numerous clusters which are tossed about the drive in no apparent order.  When a file is called up to be read the drive spins (unless solid state) to assemble the file which creates heat, wear and tear, etc which all shorten the life of the drive.  Solid state drives are not excluded from fragmentation of course.  Diskeeper keeps your files defragged both in read and write mode and places your fave files and folders at the front of the drive (it does a lot more).  It runs real time, small footprint and no performance impact.  All HD drives on a single PC are kept at 0% fragmentation.  You can run all kinds of reports on the drive.  There are various indicators that will tell you the drives health as well.  Images / Videos due to their size (IMHO) have the one of the biggest impacts on a drive's performance.  I minimize my risks with both of these software tools and have yet to have an issue losing with my personal data or a client's.  I've got a link to disk keeper on my site if anyone is interested drop me a note.  I can get you a demo. PS they are now called Condusiv Technologies.  Sorry if this sounded like an ad but its my personal experience and using these products daily.  Been using Diskeeper for approx. 9 years and know it well.  It's on every server I manage.

I've used diskeeper in the past, worked fine. Generally if it's a single large disk that  you simply dump photos onto without deleting, chances are you won't get much, if any, fragmentation. Paragon seems like simply a quality backup solution, but there's nothing inherently unique about it. Doing basic S.M.A.R.T. checks, there are a bunch of tools out there, and you can always use something like rsync and/or par2 parity blocks and checks which are F/OSS. However, Paragon is likely to do all that in a much easier to use, nicer to use, prettier package that will probably be well worth the cost since it seems fairly inexpensive.

That said, data at rest on a HDD is, over time, likely to experience bit-rot, which is where a proper backup solution will include some form of parity generation which can at worst detect a problem, and most of the time actually fix/recover the missing data. This you get automatically with a filesystem such as ZFS, but not NTFS, Fat32, Ext2/3 (not sure about Ext4, I don't think so). You can generate that information from my previously mentioned par2 generated parity blocks, or I'm sure there are other tools out there. Not sure if Paragon would do that or not, something to look for in the feature list anyway.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #36 on: November 14, 2012, 01:32:12 PM »

dirtcastle

  • Canon 6D
  • *****
  • Posts: 391
    • View Profile
    • Eric Nord Flickr Page
Re: Best Methods For Long Term File Storage ??
« Reply #37 on: November 14, 2012, 03:02:21 PM »
f

Certainly, storing them online is a poor answer, companies go out of business, their systems fail, you forget to pay your bill, not a 50 + year solution at all.  In about 2000, my hard drive crashed and everythinng was lost.  I had a one year old backup on CD of my images, and recent backups on floppy disks.
 

While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive. We've all had personal hard drives fail. I've also had hosted drives fail too. But the difference was that my online host had a backup system and restored everything without me paying any extra money or doing any extra work.

Ultimately, the best strategy is redundancy. For a home solution, it means arrayed drives (and a fireproof safe). But the level of cost and maintenance on such a system is formidable and you will be doing all of the work of maintenance, organizing, and syncing. And then, presumably, you will need to completely rebuild the system every 10 years or so. And without an associated online solution, you have only local redundancy.

For me, the big difference between a home solution and "hosted" solution is that the host will upgrade their equipment over the years. They will transfer your data and maintain backups. I would never suggest relying exclusively on a cloud host. Never. But let's recognize that storing data is what these companies do; and setting up an equivalent system at home is unrealistic for most photographers.

That said, as someone who has had hosted drives go down, I can understand people's wariness of using an online solution. And it's true that if the host goes down, those crusty backups will be the last line of defense. That was why, for super critical collections, I suggested having two online hosts.

It's all about redundancy and multiple points of failure. No single system will give a high level of reliability. All of these systems involve risk.

cayenne

  • 1D X
  • *******
  • Posts: 1212
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #38 on: November 14, 2012, 03:41:53 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.
Well, you know there are other OSes out there besides Windows....some are VERY robust, etc.

You've heard the saying...the right tool for the right job.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

A little DIY and learning a new thing...is never a bad thing.

I don't use windows much, unless a tool I need to use requires it...but I figure every OS has its uses....best not to get too dependent on any one, especially one owned by a company that can change its mind, direction or go out of business as your primary OS.

You might get to start familiarizing yourself with Linux or other variants of Unix type OSes....they can really be your friend. Sure there is a learning curve.....but there is always a LC when learning any new, valuable skill, no?

 ;D

cayenne

cliffwang

  • Canon 7D MK II
  • *****
  • Posts: 491
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #39 on: November 14, 2012, 05:09:55 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.
Well, you know there are other OSes out there besides Windows....some are VERY robust, etc.

You've heard the saying...the right tool for the right job.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

A little DIY and learning a new thing...is never a bad thing.

I don't use windows much, unless a tool I need to use requires it...but I figure every OS has its uses....best not to get too dependent on any one, especially one owned by a company that can change its mind, direction or go out of business as your primary OS.

You might get to start familiarizing yourself with Linux or other variants of Unix type OSes....they can really be your friend. Sure there is a learning curve.....but there is always a LC when learning any new, valuable skill, no?

 ;D

cayenne
I used to use Linux system and built FreeNAS years ago.  However, Windows systems are very stable nowadays.  Now what I am looking for is a compromised solution between performance, simple, and few machines.  What I might going to do later is building a Windows Server 2012 with ReFS system and run some VMs on it for different jobs.  VHD will let me manage storage easily.  I will play around and to see if I need to have an extra file server or not.  Since my current file server is already 4 years old, I might build a FreeNAS next year if necessary.
Canon 5D3 | Samyang 14mm F/2.8 | Sigma 50mm F/1.4 | Tamron 24-70mm F/2.8 VC | Canon 70-200mm F/2.8 IS MK2 | Canon 100mm f/2.8 Macro L | Canon Closed-up 500D | 430EX | Kenko 2x Teleplus Pro 300 | Manfrotto Tripod

Halfrack

  • Canon 7D MK II
  • *****
  • Posts: 480
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #40 on: November 14, 2012, 05:11:58 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

If nothing else, go look at my posts about using freeNAS on this thread. It is close to turnkey...OS fits and boots from a USB thumb drive...has ZFS, and everything you'd need.
Good tutorials written/video out there on the site....

cayenne

ZFS applies to Windows very well - it's over the network, same as Windows Server 2012 (you're not going to use 2012 as your desktop OS are you?).  WinFS was supposed to be the end all, but it got dropped.

Too bad there are minimal if any options in the small form factor boxes that hold 4-6 drives.  If you have been doing giant servers, your wife is correct in making you downsize - it's for your own good :D
"Me owning a lens shop is kind of like having an alcoholic bar tender." - Roger Cicala

etg9

  • PowerShot G1 X II
  • ***
  • Posts: 45
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #41 on: November 14, 2012, 05:47:26 PM »
NAS (Raid 5 or 6) to Cloud is what I safe is the safest bet. That covers corruption, accidental file damage, disk failure, enclosure failure, catastrophic damage, and regional damage. International damage would not be covered but if the US is gone then I don't need my pictures anymore.

I have a Synology NAS at home and love it. It's small and quiet and has lots of options. I have it keep 5 snaps of any changes so I can revert if need be. There are cheaper things to buy, but their are tradeoffs and this one fit my needs. Pick a provider with a  good long term track record and replace the NAS every 4-5 years

After that data gets sent off to the cloud for an offsite backup. I use Amazon Glacier now but have jumped around and another service may fit your needs better. I copy compressed JPEG only to Skydrive as a complete last ditch about every quarter.

My photo backup is currently about 500GB (I don't keep too many raw) so that's only $5/mo. To get it all back out of there would be about $60, for DR this is cheap and Amazon is a good trusted name.


TrumpetPower!

  • 1D Mark IV
  • ******
  • Posts: 951
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #42 on: November 14, 2012, 06:39:03 PM »
Lots of questionable advice, here.

First, do not rely on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever does own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.

If you're not overly worried about your cloud provider accidentally or intentionally sharing your data with the world, it can make a nice additional supplement to your data archiving strategy, but only as a "if everything else I'm actually relying upon goes tits-up, I'd hopefully still be able to get to it in the cloud" sort of last gasp hope.

With that out of the way, the only reliable method is to continually keep all your data readily available and online and part of your regular backup routine. As others have pointed out, old media die in lots of different ways. If you copy all your old media to whatever you're using today, you don't care if the old stuff dies for whatever reason, and you're also confident that you've got a valid copy. There's no worry that your several-year-old DVDs might be starting to delaminate, or that your Zip drives will have the click-o-death, or whatever.

Yes, that means you need bigger hard drives today, but the good news is that hard drives are dirt cheap compared to whatever you spent on your old media. A single DVD doesn't even store as much as a typical CF card. A hundred bucks gets you a hard drive that holds the equivalent of a few hundred DVDs. When that drive fills up, get another.

The simplest and most reliable backup method these days is to get three times as much disk space as you need. Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.

Also worth investigating, depending on your performance needs and your desire for tidyness, are RAID arrays. Be careful; many commonly-used RAID modes actually put you more at risk for data loss than a single hard drive, meaning you need that much more redundancy in your backups to compensate. Safer RAID modes eat up more disk space. Duh! But you only want to think about RAID if a single disk isn't big and / or fast enough to hold all your stuff, and you should then think of the RAID array as a single disk that happens to have some extra moving parts.

IOSafe also makes near-indestructible hard drives: fireproof, waterproof, crushproof. They're more expensive than a regular hard drive, but very reasonably priced. If you're on a Mac, just get one (or three or however many you need) and point TimeMachine to it (them), and the only reason you'd need an offsite backup is if you're worried about theft.

Cheers,

b&

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #42 on: November 14, 2012, 06:39:03 PM »

squarebox

  • EOS M2
  • ****
  • Posts: 236
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #43 on: November 14, 2012, 07:15:13 PM »
Amazon glacier starts to get expensive as you add more data.  500GB of storage seems to work out to $5 USD a month.  I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month.  Just ftp all my pictures up there.  The only limitation is you can't have more than 250,000 files.
5Dmk3 - 16-35L - 24-105L - 70-200L F2.8 mk2 - 35L - 50mm 1.4 - 100L - 600xt-rt

Leadfingers

  • PowerShot G1 X II
  • ***
  • Posts: 43
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #44 on: November 14, 2012, 07:31:18 PM »
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium.  On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

When I hear stories of people backing up to DVD, the first thing I always think is, oh my god, how much free time do you have?  I have 140 gigs of pictures and I'm sure that's a lot less than a true pro.  Backing that up to DVD would require 28 disks for crying out loud.   

Drobo makes a pretty clever solution for all this.


As to the suggestions for FreeNAS...I haven't used it in two or three years, but when I did, I thout it was a debacle.  It was far too complicated and required far too much user configuration.  I think this was back in the version 7 days...has it improved since then?
« Last Edit: November 14, 2012, 07:38:04 PM by Leadfingers »
~ I look around me.  I describe what I see.
~~ Daria

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #44 on: November 14, 2012, 07:31:18 PM »