October 23, 2014, 03:38:15 PM

Author Topic: Best Methods For Long Term File Storage ??  (Read 17405 times)

dirtcastle

  • Canon 6D
  • *****
  • Posts: 391
    • View Profile
    • Eric Nord Flickr Page
Re: Best Methods For Long Term File Storage ??
« Reply #45 on: November 14, 2012, 07:59:20 PM »
Anyone know how various stock image companies store their data?

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #45 on: November 14, 2012, 07:59:20 PM »

Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • *********
  • Posts: 8874
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #46 on: November 14, 2012, 08:02:05 PM »

 

While it's true that many online solutions could be risky, I would trust the lifespan of Amazon over a consumer-grade hard drive.

\
You are aware that some lost everything in the recent Amazon crash??  Some lost their files when their online photo services folded.  Online storage has not really been proven to be a reliable system that you can upload to and expect to have your data 50 years from now.  Nor is a hard drive, raid system, SSD, memory card, DVD, or CD.
You must actively keep multiple backups and move data as media becomes obsolete.  That was my point, nothing is as reliable as those 100 year old prints that many of my cousins have in a shoebox in their closet.  Some may be lost, some may burn up, but there seem to always be someone who managed to save them.  I guess thats a case of many many backup copies to a media that does not lost the images after 50 or 100 years.
http://www.businessinsider.com/amazon-lost-data-2011-4
 
My Grandfather from around 1905.  You had to sit very still then, no fast lenses or shutters.
 

 
My Grandmother born in 1886

 
The point is that we still do not really have media that can match the longevity of the old prints from 100 years or more ago, except to make and store prints.

brianleighty

  • Canon 70D
  • ****
  • Posts: 267
    • View Profile
    • Leighty Photography
Re: Best Methods For Long Term File Storage ??
« Reply #47 on: November 14, 2012, 08:21:10 PM »
You do have a point but that outage was with EC2 not glacier. I'm not sure what the promised durability for EC2 is but glacier is very high due to the fact that it's stored in multiple facilities and multiple hard drives within that facility. I have not had faith to offer retaining data to my clients but Amazon's Glacier looks promising and relatively affordable. I'll of course keep a local hard drive with the data on it. But as you say, that's not enough. With Glacier, I can feel secure that even if my house burns down I'm ok. And I don't have to worry about trying to put data in other locations other than on Glacier.
Canon 5D Mark II, 50D, XSi, 24-105L IS, Sigma 35 1.4, Canon 40 2.8, Canon 35 2.0, Sigma 10-20, Tamron 17-50, Canon 50 1.8, 580 EXII, 430 EXII

dirtcastle

  • Canon 6D
  • *****
  • Posts: 391
    • View Profile
    • Eric Nord Flickr Page
Re: Best Methods For Long Term File Storage ??
« Reply #48 on: November 14, 2012, 08:23:09 PM »
@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.

JerryB

  • Guest
Re: Best Methods For Long Term File Storage ??
« Reply #49 on: November 14, 2012, 09:24:52 PM »
It might be the only effective way to save our photos is to print them out.  Hard drives fail, DVD's, and CD's only last so long, and who knows if any of these will exist in a few years.  But prints will still be there. 
Many companies use RAID array's which consist of multiple drives to backup their files.  Some like, credit card companies, have a backup of their array's located halfway around the world.

tron

  • 1D X
  • *******
  • Posts: 1862
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #50 on: November 14, 2012, 09:26:29 PM »
Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
First person answers: "I also have a monthly card"  ;D

Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...

bkorcel

  • Guest
Re: Best Methods For Long Term File Storage ??
« Reply #51 on: November 14, 2012, 10:43:44 PM »
Now that you mention it, Many safes now are coming with integrated power and USB hubs so you can lock up your hard drives or small servers inside to protect from theft.  Some are even fire rated.

Someone goes on a bus and validates two tickets instead of one! So a second person approaches him and asks him why he has done that.
First person answers: "In case one of the tickets is lost". But then the second person says: "Well, in that case you may as well lose both tickets".
First person answers: "I also have a monthly card"  ;D

Well this joke reflects my opinion for the use of hard drives. I store my photos in many copies. I have not yet filled another disk to put it in a safe box though...

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #51 on: November 14, 2012, 10:43:44 PM »

cayenne

  • 1D X
  • *******
  • Posts: 1212
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #52 on: November 14, 2012, 11:01:46 PM »
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium.  On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.

When I hear stories of people backing up to DVD, the first thing I always think is, oh my god, how much free time do you have?  I have 140 gigs of pictures and I'm sure that's a lot less than a true pro.  Backing that up to DVD would require 28 disks for crying out loud.   

Drobo makes a pretty clever solution for all this.


As to the suggestions for FreeNAS...I haven't used it in two or three years, but when I did, I thout it was a debacle.  It was far too complicated and required far too much user configuration.  I think this was back in the version 7 days...has it improved since then?

freeNAS looks pretty straightforward from what I'm seeing.

I'll try to report back when I get the box constructed and running....

It is version 8.x.x something now.....

C

brianleighty

  • Canon 70D
  • ****
  • Posts: 267
    • View Profile
    • Leighty Photography
Re: Best Methods For Long Term File Storage ??
« Reply #53 on: November 14, 2012, 11:42:29 PM »
For those that use Glacier, have you actually had to pull any data off? Reason I ask is their retrieval charge is astronomical if you don't spread it out correctly. Without some special software to this, I think this ones dead in the water for me now :(
Canon 5D Mark II, 50D, XSi, 24-105L IS, Sigma 35 1.4, Canon 40 2.8, Canon 35 2.0, Sigma 10-20, Tamron 17-50, Canon 50 1.8, 580 EXII, 430 EXII

etg9

  • PowerShot G1 X II
  • ***
  • Posts: 45
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #54 on: November 14, 2012, 11:50:19 PM »
As an IT guy who designs DR (just a part of what I do) for a living and like I said earlier in this thread a working copy on site, a back up on site and a backup offsite and in many locations as possible is the safest bet.

I never told anyone to use the cloud as a single backup method. Tape is a poor backup solution for long term, as are DVD's but for different reasons. DVD's tend to get scratched or wear out, tapes no one has that model tape drive 10 years later and they are not backwards compatible. Single external hard drives die all the time.

Keep you data yourself, on a NAS. Use the cloud as a backup to that. Very easy to manage and very hard to kill.

Print is a bad idea for archival, I'm happy MSP's photos are around. a lot of people's aren't: fires, floods, tornadoes, earthquakes, burglars, or just getting lost have claimed lots of photos. They also take lots of room and are hard to move about and organize. This may work as a last ditch backup of very important family photos.

People telling you to keep backups in bank boxes or at parents houses tend to live close by, no geographic dispersion. that big earthquake/flood/fire may very well hit them too. 



do not rely on Teh Cloud, in any form. If you don't have physical control over your data, you don't own it, and whoever does own it can do anything they want to it and you've got very little recourse. That could include deleting it, peeking inside it, or even sharing it with the world.

Disk(s) 1 are where you keep everything. Put a copy of everything on disk(s) 2. Every week (or month or whatever), take disk(s) 2 offsite to your bank deposit vault or your parent's place or somewhere you trust and exchange it for disk(s) 3, which you bring back with you and start treating as you used to do with disk(s) 2. The next week, do the swap again.

But you only want to think about RAID if a single disk isn't big and / or fast enough to hold all your stuff, and you should then think of the RAID array as a single disk that happens to have some extra moving parts.

...and the only reason you'd need an offsite backup is if you're worried about theft.


The Cloud, read the TOS for the provider and make sure it fits your needs. don't make the cloud (or anything) your single backup source.

The disk swap is a ton of work, hopefully the last backups were good and hopefully the disk didn't fail. when you fill up one disk you get to buy 3 more. don't do this.

RAID (other than 0-1) also protects against hardware failure with an extra hot spare (5/6) or array (10/50). A NAS also doesn't have to be a single drive, the space can be logically separated.

or any other natural disaster.



JaxPhotographer

  • PowerShot G1 X II
  • ***
  • Posts: 35
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #55 on: November 15, 2012, 12:17:22 AM »
I currently use RAID 1 (mirroring) on 2x2TB Drives in my primary computer just for image storage. I have SSD's for my OS and primary Apps and non-redundant HDD's for non-critical files.

I keep an active 3 TB External connected for nightly scripted backup of the RAID 1 files.

I keep another 3 TB External at work that I bring home once a month to backup and store as my off-site.

I replace all my drives every 2-3 years to reduce the risk of aging drive failures.

In the event I have any failure in either drive in the RAID 1 array, the other is still fully functional. I perform an immediate backup to the external, grab a spare 2TB spare, and swap the problem drive out and allow it to automatically rebuild the mirror image on the new drive.

Perhaps at some point I may consider cloud but have found that my current system has served me well and survived a hard drive failure once already. If I am able to increase client level work, then I will probably make the move to cloud storage as my off-site method.
5DMKIII | 7D | 16-35L f/2.8 II | 24-105L f4 | 70-200L f/2.8 IS I&II | 50 f/1.8 | 85 f/1.2 II |1.4x & 2x Ext | 100mm f/2.8L IS | MT-24EX Macro Twin Lite | 2x600EX-RT | ST-E3-RT | 430EX Speedlite | Sekonic L-758DR | Induro CT314 Tripod | Arca-Swiss Monoball Z1 dp  |

gmrza

  • Canon 7D MK II
  • *****
  • Posts: 497
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #56 on: November 15, 2012, 01:56:49 AM »
Lots of VERY questionable advice here in my opinion.

By far the easiest solution is to get a raid5 NAS box and use it as your primary storage medium.  On a regular schedule you can make a redundant copy to an external drive and store it somewhere else.


One point to take into account especially with RAID5 or RAID6 - you probably want to make use of enterprise drives (not desktop drives) to be sure that you get the reliability you want.
Zeiss Ikon Contax II, Sonnar 50mm f/2, Sonnar 135mm f/4


canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #57 on: November 15, 2012, 01:59:48 AM »

el bouv

  • Power Shot G7X
  • **
  • Posts: 14
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #58 on: November 15, 2012, 02:13:13 AM »
You could also consider Thecus N7700 series.

Refer to http://www.thecus.com/product.php?PROD_ID=14

It basically seems to be Freenas with a reasonable management GUI in a 7 slot box with support for both Apple and PC and iSCSI et al..

Supports RAID 5 and 6 and now has a 10GE interface.

I run a 7x2TB and a 7x3 TB setup which yields 8TB and 12 TB formatted data space.  I prefer RAID 6 as it can take more than 12 hours to rebuild a failed array. This puts a RAID 5 setup at risk during the rebuild window.

The Thecus does only have 1 power supply, so there is no redundancy and hence a single point of failure, but it has been good enough for SOHO use. In three years I have not had any failures except for drives.

I prefer normal Seagate drives, not the more expensive Enterprise quality, they last between 14000 and 18000 hours, that is between 1.5 and 2 years, and the price is OK. In my setup WD has had recurring failures in the first 1000 hours.
 
Swopping drives are easy. The system is hot pluggable so you simply unplug the drive carrier, loosen the four screws that hold the failed drive, replace drive, refasten and plug back. The system detects the new drive and functions in Degraded mode until fully rebuilt. No real impact on user experience.

PeterJ

  • Canon 6D
  • *****
  • Posts: 342
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #59 on: November 15, 2012, 02:37:13 AM »
Lots of talk of RAID but I hope everyone makes multiple backups in different locations, I'm pulling this number out of my backside but I suspect RAID protects against under 50% of real-life data loss. That includes viruses, application errors, user error, fires, lightning strikes and theft etc. Can also make recovery harder if you can't replace your EOL RAID controller with something compatible.

Personally I'd only use hardware RAID for performance or where 100% uptime was needed, otherwise spend the extra cash on extra drives for backup and leave a few you only rotate once a year or so in case you need to recover files from an earlier point.

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #59 on: November 15, 2012, 02:37:13 AM »