April 19, 2014, 06:38:53 PM

Author Topic: Best Methods For Long Term File Storage ??  (Read 13720 times)

dawgfanjeff

  • Canon AE-1
  • ***
  • Posts: 64
    • View Profile
    • Flickr Site:
Re: Best Methods For Long Term File Storage ??
« Reply #75 on: November 15, 2012, 02:15:08 PM »
I don't love the printing idea; however...all those pictures will be the most valuable thing you have when the zom-pocalypse occurs!   Except bullets of course  ;D
5DIII, | 100 2.8 Macro | 24-105L |
70-200 f/4L IS | 50 1.4 | 10-22 (for sale!)

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #75 on: November 15, 2012, 02:15:08 PM »

dirtcastle

  • 7D
  • *****
  • Posts: 391
    • View Profile
    • Eric Nord Flickr Page
Re: Best Methods For Long Term File Storage ??
« Reply #76 on: November 15, 2012, 02:19:01 PM »
@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.
I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years.  Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

Agreed!

etg9

  • PowerShot G16
  • **
  • Posts: 45
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #77 on: November 15, 2012, 02:29:05 PM »
@ MSP

I totally agree with you on the facts. Cloud storage has risks. Local storage has risks. And print is a great format, with a long shelf life.

With regards to "prints as backup storage", I think it boils down to volume. The problem with print storage is that it has scaling issues. For smaller collections (or people with lots of money), print is a great way to preserve and backup photos. But for most shooters, print doesn't work as a primary method of cold storage. It requires one to have both a digital storage system and an analog system. It's just too much work/money for most of us.

Now, having said all that... for selected images, having prints makes a lot of sense.
I'm not suggesting prints as a solution for the reasons you mention, just pointing out that with digital media, we do not have a nice reliable long term solution similar to the store it in a shoebox method so our descendents can view images 100 years from now.
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years.  Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

Agreed!

We just really have no idea, 100 years ago they probably weren't sure the prints would be around now as the first prints in the mid 1800's didn't hold up so well. RAW is a little funkier as it has to get processed when opened but JPEG is a much more developed and solid standard. I can imagine that will be around in 100 years as it's already been around for 20 and it doesn't seem like it's going anywhere.

thedge

  • PowerShot G16
  • **
  • Posts: 6
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #78 on: November 15, 2012, 04:58:36 PM »
Amazon glacier starts to get expensive as you add more data.  500GB of storage seems to work out to $5 USD a month.  I use 1and1.com as they hook me up with webspace, email addresses, and UNLIMITED webspace for just $7/month.  Just ftp all my pictures up there.  The only limitation is you can't have more than 250,000 files.

FYI, if you read the terms of service for 1&1 (and most or all webhosts), using the FTP space as a backup is not allowed. Them finding out is different, but it is against the ToS.

mws

  • Rebel T5i
  • ****
  • Posts: 182
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #79 on: November 15, 2012, 05:30:22 PM »
Not by any means a feasible solution, but I wonder how many sheets of paper I would need to print out all my photos in a series on 1s and 0s………...

tron

  • 1D X
  • *******
  • Posts: 1673
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #80 on: November 15, 2012, 05:56:11 PM »
Not by any means a feasible solution, but I wonder how many sheets of paper I would need to print out all my photos in a series on 1s and 0s………...
;D ;D (poor trees)


canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #81 on: November 15, 2012, 08:36:52 PM »

Mt Spokane Photography

  • 1D X
  • *******
  • Posts: 7711
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #82 on: November 15, 2012, 08:40:59 PM »
My Great Great Grandfather who moved from a real house in Upstate New York to a sod house in Kansas in 1874.  Note that the floor was below the surrounding ground.  The first winter, it rained for weeks, so you can imagine the wet muddy mess inside.
This is from a newspaper clipping, I haven't been able to track down the actual print.  Those sod houses were replaced with log houses as soon as possible.  It looks like they were laying out a new house from the dug out ground and the string and stakes on the lower right.  I have a diary telling the story of that move.
If we suceed in being able to save our photos for 120 years, they will be a far cry from this one.  However, its entirely possible that the glass plates for this one are in a university library or a  museum somewhere in Kansas.
 

bkorcel

  • Guest
Re: Best Methods For Long Term File Storage ??
« Reply #83 on: November 15, 2012, 09:44:53 PM »
The biggest problem I see with this idea is that after 100 years no one will remember what USB is or have anything that will read it!  You'll have a wonderful silver paper weight.   Kind of like all of these 3 1/2"  drink coasters I have now.

http://www.sandisk.com/products/usb/memory-vault/

Drizzt321

  • 1D X
  • *******
  • Posts: 1645
    • View Profile
    • Aaron Baff Photography
Re: Best Methods For Long Term File Storage ??
« Reply #84 on: November 16, 2012, 12:37:56 AM »
The biggest problem I see with this idea is that after 100 years no one will remember what USB is or have anything that will read it!  You'll have a wonderful silver paper weight.   Kind of like all of these 3 1/2"  drink coasters I have now.

http://www.sandisk.com/products/usb/memory-vault/



Even beyond that, the real information on how it keeps your data from bit-rot via cosmic rays and such is non-existent. What type of parity/crc protection is built in? Is it using SLC or MLC NAND flash? What kind of write durability does it have? It also doesn't have any provision for data recovery. If you have to send it off for data recovery, will they let you RMA it? This seems like mostly a marketing tool, although I believe it will _physically_ last more or less as long as they say. But even assuming USB50 is backwards compatible with this (doesn't even say if it's USB2 or USB3!), you need it to be more than physically intact, you need to know that cosmic rays, marginal soldering, etc won't have corrupted the data or rendered it impossible to read off certain NAND dies, or even that the controller will still function fine.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

And-Rew

  • Guest
Re: Best Methods For Long Term File Storage ??
« Reply #85 on: November 18, 2012, 04:17:39 AM »
@ MSP
I'm hoping that someone steps up and creates a storage media that is reliable, its certainly possible, but only available to the technically astute, and who knows if anyone could read the media in 50 years.  Its not happening because no one sees a market, or maybe there is no good known technical solution (I doubt that).

Never gonna happen in these money making orientated days.

Why would some one develop a storage medium that lasts years? They'd be out of business after a few years once every one had bought one of these long term storage devices.

It would be like buying a car that doesn't rust and comfortably does 200mpg round the town and 300mpg on a run. Not gonna happen any time soon...

Ellen Schmidtee

  • 7D
  • *****
  • Posts: 441
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #86 on: November 18, 2012, 07:30:43 AM »
I rent a locker in my gym, and keep a hard disk there. I go there often enough to rotate it.
That way you keep fit too ;D

I rented a locker because the monthly fee for the gym + locker is a lot cheaper bank safe, and has more opening hours than a bank, e.g. the bank is closed on weekends.

Then the gym owner told me the lockers are for people who actually excercise, so I started doing that as well. Life isn't easy...

thedge

  • PowerShot G16
  • **
  • Posts: 6
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #87 on: November 22, 2012, 12:12:55 PM »
And to answer the OP's question...

To me as an IT guy with data backup OCD the current best solution for longer term file storage is ZFS. Its the only solution that is reliable and has integrated file checksumming to check for bit rot, corruption, memory errors, etc. Yes you can run third party programs to do checksumming on any file system, but ZFS does it on file access in addition to a full scan whenever you want.

Significant Benefits to ZFS:
-File checksumming to check for corruption
-Scrubbing your drives, which is a complete read of every single file to check it against its checksum for corruption
-Hardware agnostic and "port agnostic", meaning a pool of ZFS drives can be moved to any other computer that can read ZFS, plugged in and read, without worrying about correct order of the drives in SATA ports (as with most hardware RAID solutions) or cross vendor compatibility (aka, an Areca RAID array cant be read on an LSI card), with ZFS you only worry about the pool software version which is easy to work with (backwards compatible and pool version can be upgraded etc)
-Immediate detection of failing hard drives, cables, interfaces, etc, far earlier than most solutions so far in my experiences with failing hard drives
-Resilvering a pool (aka, rebuilding an array) is based on data on the array, not the entire thing, so a ZFS pool that has 20TB capacity but only 2TB of data only has to rebuild the 2TB of data when a drive fails vs hardware RAID which rebuilds the 20TB including 18TB of nothing
-Copy on write, which leads to (among many other awesome things) snapshots of the data, which is fantastic for back ups (oops, deleted that file, but can be copied out of the snapshot that was automatically taken 15 minutes ago, or the one from an hour ago, two hours ago, or a day ago, or a week ago, or a month ago, or a year ago)
-Filesystem is always consistent, there is no file system corruption if the power cord is yanked out or some other sudden failure
-ARC and L2ARC (Adaptive Read Cache and Level Two Adaptive Read Cache) caching, which means that a file that is read from ZFS will get stored in the servers RAM (ARC) or on a designated SSD for faster access in the future

I could go on but those are the most of the juiciest ones.

I have all my images and my Lightroom catalog stored on my NAS which is running OpenSolaris and ZFS. Yes there is a performance hit for having the catalog over the network, but with ZFS's ARC and L2ARC it is quite mitigated. If I power the server off (which clears out the ARC) and power it up, then open my catalog there is a noticeable slowdown as it is read from the disks. Then after that it is faster as the parts of the catalog that are being read regularly are sitting in the RAM on the server. Same with previews, its slow the first time a folders previews are loaded then fast after that, even after Lightroom is closed and reopened since they are still in ARC.

ZFS is not perfect, it does need some computer knowledge to make use of it. Its other downside is it is not well suited to running on crap old hardware. It needs a 64 bit CPU and 4GB RAM to start. More RAM is better. It wont work well on that old Celeron in the basement for example. But barring that, its pretty fantastic.

For the curious people, the specs on my ZFS NAS are:
Intel Core i3-2100 3.1Ghz
16GB ECC RAM
Supermicro X9SCL motherboard
Three M1015 cards, each with 8 SATA ports (yes, 24 ports total)
2x 16GB SSDs as mirrored boot drives (aka, RAID 1)
4x 300GB Raptor 10,000RPM drives in striped mirror (aka, RAID 10)
2x 50GB SSD as write cache drives
1x 120GB SSD as L2ARC read cache
8x 500GB Seagates in RAIDZ2 (aka, RAID 6)
8x 1TB WD Scorpios in RAIDZ2 (aka RAID 6)

Yes, a lot of disks. The four 300GB Raptors and two 50GB SSDs are storage for my VMWare server. The eight 500GB drives store my photos, Lightroom catalog, previews, documents, email, backups, etc. The eight 1TB Scorpios I bought as cheap refurbs to store my many TB of downloaded TV shows and such until drive prices come down and I can upgrade with 8-10 3TB drives.

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #87 on: November 22, 2012, 12:12:55 PM »