October 31, 2014, 03:59:23 AM

Author Topic: Best Methods For Long Term File Storage ??  (Read 17613 times)

Halfrack

  • Canon 7D MK II
  • *****
  • Posts: 483
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #15 on: November 13, 2012, 02:46:47 PM »
http://aws.amazon.com/glacier/

Glacier is an amazing service, literally, in the cost of backing up huge amounts of data 'just in case'.  The catch is all in the amount of time it takes to upload all of what you currently have, depending on your internet connection.  It'll cost you a fair amount to get the data back, but compared to 'never' it's damn cheap.

The dual drive solution is an easy step 1 for most folks.  If you're looking for a cool tidbit, Sentry Safe makes a few models that have a USB2 pass thru - so the drive can be in the safe and yet online at the same time.  The key to these 'data rated' safes is that the internal temp is much lower than other safes, and it's rated to how long it will keep the insides under 120 degrees.

Tape is a funny medium.  It's amazing in that you don't have to worry about motors and stuff like with hard drives, but the drive itself is a huge expense.  It is the only method to look at that has 20+ year archival bits - that and MO or UDO drives and media (small, expensive).

It comes down to why are you saving the images. 

Kids/family/vacations?  Dual hard drives, maybe toss some onto USB flash memory cards (or old 1/2/4gb SD/CF cards - great way to justify faster/bigger/new cards).  Lots of these are online as part of your smugmug/flickr/picasa/etc gallery right?

Paying work for clients?  This is where you need to consider multiple mediums and methods.  Online may not be acceptable, so you will want something that's plugged in and running (a NAS with redundancy (RAID) and a good filesystem (ZFS)), plus a copy that's not spinning.  Depending on the price point charged, having it archived to tape may be a safe cya, or encrypt it and send it up to glacier.
"Me owning a lens shop is kind of like having an alcoholic bar tender." - Roger Cicala

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #15 on: November 13, 2012, 02:46:47 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14796
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #16 on: November 13, 2012, 03:13:56 PM »
I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   :o

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Makes sense, thanks!
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Dylan777

  • Canon EF 400mm f/2.8L IS II
  • *******
  • Posts: 4217
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #17 on: November 13, 2012, 03:36:23 PM »
Always keep your files duplicated.  Any kind of media is fine.  However, you have to ensure the files you backed up is always good.  DVD is not my suggest.  Nowadays external drive is very cheap, so you can get 3TB or bigger external drive to backup your files.  Do not waste your money to buy too big external drive because hard drive price drops very fast.
Here is my setting.
* 6 2TB hard drives @ RAID 5 for master drive
* 1 3TB external drive for backing up important files
* Using MS SynToy to keep my files always updated.

+1....I used to pickup 2-3TB drives, but I now I going for min 5TB. It's getting cheaper everyday though. Keep one at home and other at my in law house with monthly backup.
« Last Edit: November 13, 2012, 03:53:10 PM by Dylan777 »
Body: 1DX -- 5D III
Zoom: 16-35L f4 IS -- 24-70L II -- 70-200L f2.8 IS II
Prime: 40mm -- 85L II -- 135L -- 200L f2 IS -- 400L f2.8 IS II

bkorcel

  • Guest
Re: Best Methods For Long Term File Storage ??
« Reply #18 on: November 13, 2012, 03:50:22 PM »
More important is not the disk longevity but after 5 years when your DV burner stops working, you will buy a new one and there is a very high probability that it will not ready your DVD's.  I've had instances where it can read sectors close to the spindle but as you move out, it gets out of spec.

I do not rely on CD or DVD media anymore.  I keep my RAW and processed files duplicated on external USB 3TB drives and my processed files go out on the cloud.  All originals get copied to at least two locations before I format the cards in the camera.

I've had an instance where a USB drive failed....because they do!  Fortunately under warranty and I was able to robocopy the original onto the new drive.

Also note that magnetic media does not last forever either.  Every few years you will need to dump the contents onto another disk, reformat, dump them back.

The cloud is my final last resource to prevent loss of processed images.



I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   :o

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Halfrack

  • Canon 7D MK II
  • *****
  • Posts: 483
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #19 on: November 13, 2012, 03:53:05 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.
"Me owning a lens shop is kind of like having an alcoholic bar tender." - Roger Cicala

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14796
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #20 on: November 13, 2012, 03:56:06 PM »
...I know it's not forever.

But what is?  I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy?  ...dating myself again, I know).  The data may be intact, but I have no way to read them...
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

ahab1372

  • Canon 6D
  • *****
  • Posts: 327
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #21 on: November 13, 2012, 04:18:04 PM »
...I know it's not forever.

But what is?  I've probably still got files stored on 5.25" floppy disks (and maybe an 8" or two, remember when things called floppy were actually floppy?  ...dating myself again, I know).  The data may be intact, but I have no way to read them...
Common problem, and a huge one for archives, libraries etc.
Every time we upgrade or dispose of hardware and/or software, we should be checking if there is data somewhere that we should be converting or copying. I'm not obliged to archive everything, and most files that I haven't looked at in a year or two, or was created with software that I cannot run anymore, are probably not worth keeping. But for pictures I might judge differently.

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #21 on: November 13, 2012, 04:18:04 PM »

cliffwang

  • Canon 7D MK II
  • *****
  • Posts: 491
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #22 on: November 13, 2012, 06:03:18 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.
Canon 5D3 | Samyang 14mm F/2.8 | Sigma 50mm F/1.4 | Tamron 24-70mm F/2.8 VC | Canon 70-200mm F/2.8 IS MK2 | Canon 100mm f/2.8 Macro L | Canon Closed-up 500D | 430EX | Kenko 2x Teleplus Pro 300 | Manfrotto Tripod

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Best Methods For Long Term File Storage ??
« Reply #23 on: November 13, 2012, 06:39:06 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

cliffwang

  • Canon 7D MK II
  • *****
  • Posts: 491
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #24 on: November 13, 2012, 07:25:47 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
I used to have FreeNAS as my NAS.  However, my wife complains I had too many systems.  Thus, I have to reduce my toys.  I have already phased out two systems.  We still have few systems at home.  My target is to have only one desktop(server) and two Windows 8 PRO tablets.  If I build another server, I will get trouble with my wife. ;D
Canon 5D3 | Samyang 14mm F/2.8 | Sigma 50mm F/1.4 | Tamron 24-70mm F/2.8 VC | Canon 70-200mm F/2.8 IS MK2 | Canon 100mm f/2.8 Macro L | Canon Closed-up 500D | 430EX | Kenko 2x Teleplus Pro 300 | Manfrotto Tripod

Drizzt321

  • 1D X
  • *******
  • Posts: 1674
    • View Profile
    • Aaron Baff Photography
Re: Best Methods For Long Term File Storage ??
« Reply #25 on: November 13, 2012, 08:03:57 PM »
Sync is a great thing, but it can be your undoing - a corrupted file would be a 'changed' file so the good copy is then over written.  A lot of times you can't tell if a file is corrupted until you attempt to open it.  This is a downfall of NTFS/FAT32/HFS+/EXT2/3 (not sure about 4).  ZFS does a good job with this, but would only be used on a NAS type of setup.

Too bad!  ZFS does not apply on Windows system.  That's the file system I really want for Windows environment.  ReFS is the only solution for Windows system.  I will try Windows Server 2012 next month after I return from my vacation.

You can still use tools like Par2 on Windows, which will generate the parity information alongside the regular file. This would let you get a higher level of confidence and reliability. You'll still need to scrub the files regularly to ensure the parity is still good, which ZFS has built in and you just cron it, as well as keep it on a separate, not always connected media, and then on a separate offsite location.

If you are going to have a NAS style machine at home for this (I'm guessing by your reference to Windows Server 2012), then why not have it running FreeBSD/FreeNAS or something similar that DOES support ZFS?
I used to have FreeNAS as my NAS.  However, my wife complains I had too many systems.  Thus, I have to reduce my toys.  I have already phased out two systems.  We still have few systems at home.  My target is to have only one desktop(server) and two Windows 8 PRO tablets.  If I build another server, I will get trouble with my wife. ;D

Hmm...maybe have one big honkin' machine that you run virtualized Windows on for your Windows needs? I've recently had the thought that maybe I'll do that for my Lightroom/PS needs...still haven't made up my mind on investigating that yet, but it's crossed my mind.
5D mark 2, 5D mark 3, EF 17-40mm f/4L,  EF 24-105mm f/4L IS USM, EF 135mm f/2L, EF 85mm f/1.8
Film Cameras: Mamiya RB67, RB-50, RB-180-C, RB-90-C, RB-50, Perkeo I folder, Mamiya Six Folder (Pre-WWII model)
http://www.aaronbaff.com

Promature

  • Rebel T5i
  • ****
  • Posts: 101
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #26 on: November 13, 2012, 08:54:03 PM »
I recommend Synology and WD Red Line drives.  Synology is a great "NAS" and if you get a 2 bay solution, you can add a second drive later.  Synology software is also much better thatn Windows Home Server 2008.
http://www.amazon.com/gp/product/B008JJLZ7G/ref=ox_sc_sfl_title_5?ie=UTF8&smid=ATVPDKIKX0DER
http://www.amazon.com/gp/product/B005YW7OLM/ref=ox_sc_sfl_title_6?ie=UTF8&smid=ATVPDKIKX0DER
70D, 10-22mm, 24-105mm f4, 50mm f1.8, 85mm f1.8, 70-200 f2.8, 430EXII

kdw75

  • PowerShot G1 X II
  • ***
  • Posts: 32
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #27 on: November 13, 2012, 09:00:52 PM »
Because of changing media and technology I tend to favor cloud storage. I keep a copy on my computer and when my drive fills up I put in a larger one and put the old one in a drawer. This way I can always revert back to older drives if necessary. Then I also keep my stuff stored with a backup service since keeping it in that form doesn't rely on me having a particular computer or medium to read it. It is accessible any where I go with an internet connection. I probably should backup off site, but I haven't gone that far. I plan to get a RAID system at some point.
EOS 7D & 6D  |  24-70 f2.8 L  |  60mm f2.8 Macro  |  430ex II  |  EF-S 15-85 IS USM |  70-300 f/4-5.6L IS

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #27 on: November 13, 2012, 09:00:52 PM »

gmrza

  • Canon 7D MK II
  • *****
  • Posts: 497
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #28 on: November 13, 2012, 09:14:50 PM »
I don't use any optical media (DVD/Bluray) because they become unreadable after a few years.

How many is 'a few'?  I recently listened to The Cars Greatest Hits, a CD that I bought in 1985.   :o

That's also a pressed CD, vs a burned CD/DVD. The material in a burnable CD/DVD is, generally, at least partially organic and over time degrades. Some of them are 'archival' quality, but I believe that's generally only guaranteed for only 40-50 years. Not sure, I'd have to look that number up, but I know it's not forever.

Makes sense, thanks!

A bit of reading I did into DVD/BluRay media recently basically indicated that there are no real standards for archival optical media, and quite honestly, your mileage may vary. - It is rather uncharted territory, and there seems to be no consensus as to how long a burned DVD or Bluray disk could last.  (I have some CD-Rs which are around 20 years old, and still readable. - Is that pure luck?  I have no idea.)

The bottom line is that for data backups you need to have a policy of copying to new media every few years.  That however tends to happen by default as existing drives become too small and larger, cheaper drives hit the market.

To be heretical: the most reliable long-term storage medium for a photo still seems to be paper. Provided a print is stored under controlled conditions, you may achieve the rated life of around 200 years for archival paper. - I don't think any of us would be around to confirm.  Of course, that does not answer the OP's question!
Zeiss Ikon Contax II, Sonnar 50mm f/2, Sonnar 135mm f/4

CharlieB

  • Canon 70D
  • ****
  • Posts: 302
    • View Profile
Re: Best Methods For Long Term File Storage ??
« Reply #29 on: November 13, 2012, 10:15:00 PM »
I dump to primary HD on desktop.  It automatically copies to secondary on my Windows server.  The Windows server cross-pollinates with Linux server nightly.   Done.  Three copies, three machines, three hard drives.

CD... dont last, unless you do a slow burn on them.  I've got two and three year old CD's... lots of problems with those.

canon rumors FORUM

Re: Best Methods For Long Term File Storage ??
« Reply #29 on: November 13, 2012, 10:15:00 PM »