canon rumors FORUM

Gear Talk => Technical Support => Topic started by: FTb-n on February 16, 2013, 02:55:29 AM

Title: How do you store and archive your images?
Post by: FTb-n on February 16, 2013, 02:55:29 AM
With my XT and various PowerShots, my backup strategy involved a 500 GB external drive and burning DVD's.  With a 60D and 7D, I've had to up the capacity quite a bit.  Now it's two 1.5 TB external drives and burning BluRays (25 GB per disc). 

On the external drives, which I manually mirror, I keep straight-out-of-the camera images in one directory tree and post processed images in another.  I can shoot anywhere from 500-2,000 images per event.  Admittedly, when lighting is reliably even (particularly in color), I shoot JPG.  But, as things get moe challenging light-wise, more likely to need post work, or more important, I'll shoot RAW.  25 MB RAW images add up in a hurry.  Those 1.5 TB drives have nearly 1 TB of photos.

So, as I contemplate another set of 1-2 TB drives, how do you store and archive your images?
Title: Re: How do you store and archive your images?
Post by: brad goda on February 16, 2013, 03:30:58 AM
since 2003 went digital storing all archives on DVD
by 2008 we hit DVD archive 849... and called it quits to archiving to DVD...
shoots are labeled by date then job/event... within folders contain raw proofs selected raws duplicated and final folder containing 16bit TIFF and then duplicated sized jpegs after post.
with amount of files per job/event all work is now saved on 1TB removable HD...
as the jobs/events are completed/delivered they are placed into year/months archive folders...
screen capture the files and placed in archive file for reff in admin computer..
as the archive folder fills to 85% full its pulled and set aside as archive only... we use wiebe RTX drive housing and their drive storage boxes... sata drives are now like video cassettes that are stored in water proof case stored in steel storage cabinet.
Ive kept to 1TB just for thinking that thats enough stuff on one drive... so far all is good.
for DVDs we NEVER placed stickers and only used archival pen to mark clear center area with DVDs number... all stored in tyvek in archival DVD storage boxes... the best DVDs were used and still used... preferring the printable type with heavy white printable surface... in the past ive bought cheap DVD and regretted scratching not the bottom but the top of the DVD on insertion... the thin layer which contains the data on the metallic layer rips and flakes off... scarry.. never again.
good luck hope my example leads you to your own system.
Brad
Title: Re: How do you store and archive your images?
Post by: florianbieler.de on February 16, 2013, 04:03:35 AM
On my hard drive and backed up on an external drive. Maybe I should also sync them with my laptop for a third instance just in case everything goes south.
Title: Re: How do you store and archive your images?
Post by: brad goda on February 16, 2013, 04:22:46 AM
oops I failed to mention...
system drive contains OS and software...
all image folders are on second internal drive or outboard RTF drive...

if you are using Imac or laptop duplicate all image folders to portable or storage drives...
I consider any files being saved on the system drive have more potential for loss error or corruption.
work and store all image on other drives...

if need be get it off system asap and archive on your method
Title: Re: How do you store and archive your images?
Post by: bakker on February 16, 2013, 04:43:42 AM
I had all my pictures burned to DVD's but recently I put them all back on my 1,5 TB HD. (That's my OS drive right now, though I'll need to get an SSD for OS soon..). I have the folder "Photo" synched with another 1,5 TB drive. That one used to be an external usb drive but I built it into my PC for a faster synching speed through the SATA connection.
Title: Re: How do you store and archive your images?
Post by: candyman on February 16, 2013, 04:55:57 AM
I have two external Harddiks with each 3TB diskspace. They are USB 3.0 and it's a double backup system on weekly basis of my laptop.
I am currently working on a NAS system with raid.

Title: Re: How do you store and archive your images?
Post by: mrsfotografie on February 16, 2013, 05:11:57 AM
On an internal 1TB HDD that I mirror to a 3TB HDD in a HTPC on a regular basis. The HTPC is backup and a great way to view the photos on a 40" flatscreen TV.

I have a legacy external 1TB HDD but its interface is USB 2.0 - I may convert that to E-sata (internals is sata) to get a second backup solution. USB 2 is too slow IMHO.
Title: Re: How do you store and archive your images?
Post by: Sith Zombie on February 16, 2013, 05:43:12 AM
Fairly straightforward, everything goes into aperture on my computer HD and then I do regular backups of my whole HD [via time machine] on to a 1TB external. I keep thinking of backing up to a 3rd device but to have 2 HD failures would be very rare. Also if they do fail, its usually a mechanical part that goes, so the actual data is still safe.
Title: Re: How do you store and archive your images?
Post by: gferdinandsen on February 16, 2013, 06:48:42 AM
I use several redundant methonds.

First, primary storage where I do all my editing from, 2x 512GB Solid State Hard Drive
Second a 2TB NAS in RAID 1+0 configuration
Third a 1TB external HDD
Fourth, M-Disc's (archival quality DVD storage) (http://www.mdisc.com/what-is-mdisc/ (http://www.mdisc.com/what-is-mdisc/))

Finally a use a simple script that runs as a scheduled batch job that utilises RoboCopy to copy the Change Delta's to the alternate media (except to the M-Disc's, only to the HDD's)
Title: Re: How do you store and archive your images?
Post by: FunPhotons on February 16, 2013, 07:06:29 AM
Here's how it goes for me, first insert a CF card
 

Cloud storage would be nice but its too expensive, and I have about 1.5TB of pictures at this point and growing. This also stores all of my other data, most of my life is on those disks.
Title: Re: How do you store and archive your images?
Post by: gferdinandsen on February 16, 2013, 07:19:46 AM
Here's how it goes for me, first insert a CF card
  • LR4 imports to a working copy disk converting to DNG. Backup is made to a different disk
  • To eject card I run a script which does a rsync to a 15TB (5x3TB) on a Synology 1511+ NAS with 2 disk redundancy (up to two disks can crash and recover)
  • Then it asks whether to erase and eject card (3 copies of photos at this point) which I say yes
  • Once a night everything is backed up to a Synology 'side car' with 1 disk redundancy  12TB (4x3TB)
  • Once a week everything is rsync backed up to a Drobo FS 5x2TB with 1 disk redundancy
  • Once a month everything is rsync backed up to a briefcase of disks which reside offsite at my workplace
  • An extra backup set exists on portable hard drive (3x1TB) which I take with me when traveling
  • When traveling all shots are copies to a dual disk redudancy mini RAID enclosure
 

Cloud storage would be nice but its too expensive, and I have about 1.5TB of pictures at this point and growing. This also stores all of my other data, most of my life is on those disks.



Cloud storage has gotten a lot cheaper, ASUS now offers 500GB for just $50.00/yr
Title: Re: How do you store and archive your images?
Post by: brad goda on February 16, 2013, 07:57:32 AM
Here's how it goes for me, first insert a CF card
  • LR4 imports to a working copy disk converting to DNG. Backup is made to a different disk
  • To eject card I run a script which does a rsync to a 15TB (5x3TB) on a Synology 1511+ NAS with 2 disk redundancy (up to two disks can crash and recover)
  • Then it asks whether to erase and eject card (3 copies of photos at this point) which I say yes
  • Once a night everything is backed up to a Synology 'side car' with 1 disk redundancy  12TB (4x3TB)
  • Once a week everything is rsync backed up to a Drobo FS 5x2TB with 1 disk redundancy
  • Once a month everything is rsync backed up to a briefcase of disks which reside offsite at my workplace
  • An extra backup set exists on portable hard drive (3x1TB) which I take with me when traveling
  • When traveling all shots are copies to a dual disk redudancy mini RAID enclosure
 

Cloud storage would be nice but its too expensive, and I have about 1.5TB of pictures at this point and growing. This also stores all of my other data, most of my life is on those disks.



Cloud storage has gotten a lot cheaper, ASUS now offers 500GB for just $50.00/yr

great... i have 11TB of data to store on cloud and that will cost 1100.00 per year... wow. with data growing X 1.75 per year... yikes... not for me

but I think original poster you are getting alot of good ideas...
Title: Re: How do you store and archive your images?
Post by: Chewngum on February 16, 2013, 08:14:22 AM
Look up amazon S3 cloud storage. Glacier storage is super cheap to upload to, like super cheap, and then you only pay when you download the stuff back.


There is a point where one can have successfully backup.
For me it would be:
RAID 5+0 with 2 spare disks in computer
NAS RAID 0
Offsite NAS RAID 0
Glacier cloud storage


I currently only use in computer RAID, an external hard drive and a NAS drive but if my business was a big serious thing then the above list would be sufficient though perhaps with scalability included too.
Title: Re: How do you store and archive your images?
Post by: FunPhotons on February 16, 2013, 08:24:50 AM
Uh, guys, Cloud storage isn't an option for many of us. I'm well familiar with Glacier/ASUS etc etc. Price out 4TB of data, growing at at least 1TB/Year, factor in how you'll get it to them (Cable modem isn't going to work), data availability (you still need local copies), plus redundancy and it gets expensive. Plus Cloud doesn't mean safe.

Ultimately nobody cares about your data as much as you do. There's no way a professional photographer (certainly not videographer) could make do with Cloud storage. At best it would work for a light shooting amateur.

My attitude is that data storage costs drop exponentially like all other information technologies, so I store everything and then some. In a few years I'll laugh about the day I fretted about a few TB.

Speaking of which I laugh at the folks who call high mega pixel cameras "mega pickles" (as if that's funny), and then go on to say it will never happen because "oh my god all those pixels will kill your computer and hard drive". Yeah, sure grandpa ... :)
Title: Re: How do you store and archive your images?
Post by: Mt Spokane Photography on February 16, 2013, 02:23:26 PM
I have a 6 disk QNAP NAS which backs up my PC's and images.  Its a few years old now, but has 12 TB of drives in it.  Since it runs Raid, it actually has about 9.2 TB usable.  I'm down to less than 2 TB free, so its time to clean up.
The drives can be upgraded to larger disks (12 TB Max if one actually existed), but by the time I need them, I'll likely upgrade to a new model NAS maybe with SSD.
Title: Re: How do you store and archive your images?
Post by: iKenndac on February 16, 2013, 04:06:04 PM
My photography machine has an SSD and a hard drive inside it. For day-to-day use, my Lightroom Catalog lives on the SSD for fast load speeds, and the RAW files live on the 1TB hard drive.

All of this gets backed up to an external 3TB drive for redundancy.

All of *that* gets backed up to an online provider. Being in the Land Of Fast Internet (Sweden), my upload speed makes this a breeze. I'm using a Mac application called Arq to back up to Amazon Glacier - the pricing is pretty great and Amazon's services are incredibly reliable.

People seem to dislike online storage - "just because it's online doesn't mean it's any safer". That's true, but that's not the point. The point of data redundancy is to reduce the chance of all of your copies failing at once to an acceptable level. Having a local backup to an external drive saves me from a drive failing. Having an online backup as well saves me from all of my local drives failing at the same time (very unlikely) and my house catching fire (statistically, probably less unlikely). And, let's face it — if something so catastrophic happens that causes all of my drives *and* Amazon's services to fail beyond recovery, my data will probably be the least of my worries!
Title: Re: How do you store and archive your images?
Post by: FTb-n on February 17, 2013, 01:22:39 AM
First, thanks for the feedback.  It's a tremendous help.

For those who burn BluRay, what brand media do you like?  I'm using an ASUS external drive and recently went through a 25pk of Verbatim BD-R 25GB 6x discs.  Of the 25, 8 were unusable, the drive simply rejected them.
Title: Re: How do you store and archive your images?
Post by: Botts on February 17, 2013, 09:16:14 AM
1st backup -> Daily Aperture Vault on external NAS
2nd backup -> Monthly vault on external HDDs kept at work.  Brought home only to backup.  (Currently using an OWC 4-bay enclosure)
3rd backup -> CrashPlan+ backup 1,500 miles away.

I'd consider blurays, but the cost is too high per GB right now for me.
Title: Re: How do you store and archive your images?
Post by: JaxPhotographer on February 17, 2013, 09:51:11 AM
I currently use RAID 1 (mirroring) on 2x2TB Drives in my primary computer just for image storage. I have SSD's for my OS and primary Apps and non-redundant HDD's for non-critical files.

I keep an active 3 TB External connected for nightly scripted backup of the RAID 1 files.

I keep another 3 TB External at work that I bring home once a month to backup and store as my off-site.

I replace all my drives every 2-3 years to reduce the risk of aging drive failures.

In the event I have any failure in either drive in the RAID 1 array, the other is still fully functional. I perform an immediate backup to the external, grab a spare 2TB spare, and swap the problem drive out and allow it to automatically rebuild the mirror image on the new drive.

Perhaps at some point I may consider cloud storage but have found that my current system has served me well and survived a hard drive failure once already. If I am able to increase client level work (more paid work), then I will probably make the move to cloud storage as my off-site method.

Title: Re: How do you store and archive your images?
Post by: mws on February 17, 2013, 10:26:07 AM
I have 1 TB in my laptop, that is only about half full, I back everything up to a 3 TB NAS in RAID 1 with Carbon Copy cloner. I also have a 2nd 3 TB NAS in RAID 1 in my detached garage that serves as offsite incase the house burns down. The NAS in my basement replicates itself to the one in the garage twice a week. I thought about adding another layer of backup with Amazon S3, but decided against it due to cost. I figured the two NAS was good.

My real concern is when my laptop fills up, I really don't want to keep something attached to it to use Lightroom, since LR doesn't play well with network volumes.
Title: Re: How do you store and archive your images?
Post by: viggen61 on February 17, 2013, 11:23:52 AM
I use Aperture on a Mac, and here's how I do it:

Aperture handles the vaults seamlessly, and doesn't skip a beat if one is offline. I swap the bare drives back and forth as needed. At worst, I'll lose a few months of images (yes, I'd like to lessen that, but...). Hopefully, though, I'll have at least copied the best of them up to Flickr or iCloud, so I'll have something!
Title: Re: How do you store and archive your images?
Post by: eos650 on February 17, 2013, 11:54:44 AM

I recently purchased a large safe that has both power and Ethernet inside. I'm considering moving either my file server or one of the backup drives into this safe as my computer and file server are currently vulnerable to theft, fire, flooding, etc.

Also, several of my better images are loaded at full resolution to an online site. If all of my other backups failed, I would still have access to the cream of the crop.

I don't use raid 1 (mirroring) as raid is NOT a backup strategy. Raid is a tool for reducing down time and can be very effective for this, however as others have said here, additional backups are still required. In my case raid would be an unnecessary use of resource, since should my drive fail, I could bear a couple of days of down time, while recovering from a backup.
Title: Re: How do you store and archive your images?
Post by: thedge on February 17, 2013, 03:48:51 PM
I use a NAS running a form of Solaris, giving me the ZFS file system. My catalog, previews and all photos are stored on a RAID6 of 8 hard drives (RAID5 is dangerous with large drives, RAID10 isnt really redundant enough). ZFS is discussed in another thread, but the rundown is that it has built in checksumming (prevents file bit rot) and built in filesystem snapshots. Mine snapshots the photography array every 15 minutes and keeps hourly, daily, weekly, monthly and yearly snapshots of the file system state. I can go back one year (server is about a year old, eventually ill be able to go back further) and retrieve pictures deleted at that time.

That is replicated to external eSATA drives that are rotated off site. The snapshots are replicated as well.

I don't bother with online backup. I may at some point export all my photos as jpegs and back those up online but until then I don't see it as a viable enough backup strategy given my upload speed and size of my data (both photos and other important data).

I wont ever not use RAID for important data, far too much of a risk to the data. Its a vital part of data security, just like off site backups. It is IMHO completely foolish to not use both together.
Title: Re: How do you store and archive your images?
Post by: thedge on February 17, 2013, 04:04:21 PM
First, thanks for the feedback.  It's a tremendous help.

For those who burn BluRay, what brand media do you like?  I'm using an ASUS external drive and recently went through a 25pk of Verbatim BD-R 25GB 6x discs.  Of the 25, 8 were unusable, the drive simply rejected them.

as i wrote, as soon as the millenniata BLU RAY´s are on the market i will buy them.

for the time being i use the normal verbatim (not LTH) .. never had a problem with them.
make sure you did not buy LTH media.. or make sure your ASUS drive supports them.

Quote
U.S.-based Millenniata (www.mdisc.com (http://www.mdisc.com)) today announced it will offer Blu-ray M-DISCs in the second quarter of 2013, increasing both the storage capacity and the accessibility of the M-DISC.
 
The announcement, made at the Storage Visions Conference in Las Vegas, marks a major step forward in permanent data storage solutions for businesses and consumers by making the M-DISC available in all the standard optical disc formats.
 
The other major advance in the growing world-wide acceptance of the M-DISC DVD and the forthcoming Blu-ray M-DISC is the marketing and distribution partnership with Imation Corp. announced at the Storage Visions Conference by Millenniata. Millenniata and Imation, the leading world-wide distributor of data storage products, have agreed to co-brand and distribute both the M-DISC and Blu-ray M-DISC under Imation's TDK, Memorex and Imation brands.
 
RITEK Corporation, the leading manufacturer of optical storage media in the world, will produce the new Blu-ray M-DISC as part of its manufacturing agreement with Millenniata. In addition, RITEK has signed a license agreement with Millenniata to distribute and co-brand both the DVD and Blu-ray M-DISCs through its established distribution and reseller channels as a secondary distribution channel for the M-DISC products behind Imation.
 
The new Blu-ray M-DISCs will be writable and readable on any Blu-ray combo drive - an enormous step for Millenniata and the convenience of this permanent storage technology. The Blu-ray M-DISCs will also offer at least five times the amount of storage as the standard 4.7GB M-DISC.

Quote
Whether you’re a scrapbooker, a business owner, a genealogist, a photographer, or all of the above, you can’t afford to lose your priceless memories or data. M-DISC™ is designed to last for up to 1,000 years, which means once written, your documents, data, medical records, photos, and more can be read forever.
 

Unlike computer hard-drives and optical discs (CDs and DVDs), M-DISC™ preserves and protects your files by engraving your information into a patented rock-like layer, resistant to light, temperature, humidity, and more. In fact, The U.S. Department of Defense put this to the test, and M-DISC™ was the only solution that passed. The M-DISC™ cannot be overwritten, erased, or corrupted by natural processes. Best of all, it’s compatible with any DVD player, which means you can access your data anywhere and anytime.

Those have a pretty terrible $ to GB ratio.... $3 for only 4.7GB.....
Title: Re: How do you store and archive your images?
Post by: Jamesy on February 18, 2013, 08:39:06 AM
- Current images on Win7 laptop (SSD)
- All other archived images on NAS
- NAS is backed up daily to 2TB external drive
- NAS is backed up monthly to an encrypted drive for offsites

So with the scheme above I have three to four copies of my data. I have used SyncBackPro for years and highly recommend it for Windows environments.
Title: Re: How do you store and archive your images?
Post by: Dylan777 on February 18, 2013, 09:15:15 AM
I'm not a pro...just an ave Joe's with 2 kids, who enjoy photography alot.

I'm using two 5TB ext. drives to save raw. Once the photos are done with PP in JPEG, I save them on ext drives and online Phanfare for backup and viewing.
Title: Re: How do you store and archive your images?
Post by: tomscott on February 18, 2013, 10:59:04 AM
I use a Mac Pro which has a 128gb SSD (which I keep in my spare DVD drive bay via a DVD-2.5" converter) which I run all my applications from then 4 storage drives two 1tb drives and two 2tb drives. Both drives are mirrored so they are backed up in two places incase of a drive failure. The 2tb is for my images which is nearly full and the 1tb is for general stuff.

I then have a 4TB back up drive which does an overall back up of everything. I also archive everything on DVD because if the worst happens flooding etc at least they can be retrieved only problem is they are cumbersome.

Storage is so cheap your better off being safe. I dont like having drives all over the place clutters my desk and also my brain lol! So I like to keep everything internal and have one external drive which takes care of a third back up.

3 is the magic number to safely store your data. Best thing to remember is that HDDs fail, its a fact of life they aren't very reliable, many say to swap the drives after a years use to fresh drives.

I have had a drive fail and it is the first and last time it will ever happen without a proper back up. Can be very emotional loosing your digital life. Fortunately in that instance I managed to restore some important data so not all was lost.