Storage & backup - advice needed please

[ereka]

Long time no post!

Apologies - I've been busy shooting - in fact so busy that I've almost filled up a 2 terabyte internal hard drive and a 1 terabyte external hard drive and badly need advice from you guys please as to the most reliable and cost effective way of expanding my storage and maintaining adequate backups.

OK, you'll need some background info so I'll do my best to explain in a nutshell my current set-up and workflow.

I'm a freelance photographer (occasional social photographer, but bread and butter work is currently cars and real estate).

My main computer is a Mac Pro, which I bought in 2011. It has 4 hard drive bays but currently has only 3 x 2Tb hard drives installed (a boot disc used purely for the OS and software programmes, a master disc for image storage and a backup disc). I also have 1 Tb of old images on a LaCie 2-big triple external hard drive (2 x 1Tb RAID) but I rarely need to access these and they are kept, I guess, mainly for sentimental reasons.

Recently I've had to purchase a Windows laptop because a new client's proprietary software system that I need to upload images to on a regular and frequent basis only works with Internet Explorer. The laptop is useless for photo editing as the screen is only around 75% of RGB. Therefore I don't store any images on the laptop. All image storage and processing is done on the Mac Pro. I then upload the finished images to Dropbox and from there via the laptop to the client's system.

In my mind at least, I need to expand my storage to around 4Tb with adequate backup facilities.

I work from home and we have Virgin Media 100Mb cable internet and a wireless broadband router installed, shared by 3 x Mac and 1 x Windows laptop.

Storage

Instead of buying another internal hard disc for the Mac Pro, I'm thinking that once I have adequate external backup installed I can reuse the 'backup' disc for storage, thus giving me 4 Tb of internal storage which should keep me going for quite a while.

Backup

I've considered buying an Apple Time Capsule for wireless backup but the largest unit available appears to be 3 Tb so I've also looked at various NAS storage and backup solutions but I'm getting confused by so many different options!

Workflow

I'd rather avoid the need to use Dropbox as uploading images to Dropbox is a another time consuming step in the workflow that I'd rather avoid if possible.

Budget

No more than a few hundred GB pounds!

So over to you guys - what in your considered opinion is the best storage and backup solution i.e. the most efficient, reliable and cost effective solution in my circumstances and within my budget?

 

[Marsu42]

Recently I've had to purchase a Windows laptop because a new client's proprietary software system that I need to upload images to on a regular and frequent basis only works with Internet Explorer.

Um, why not simply run IE on the MacOS with a virtual machine?

 

[ereka]

Recently I've had to purchase a Windows laptop because a new client's proprietary software system that I need to upload images to on a regular and frequent basis only works with Internet Explorer.

Um, why not simply run IE on the MacOS with a virtual machine?

I have to admit I didn't think of that

In any event though, I've always tried to keep the Mac Pro as 'clean' as possible by using it exclusively for image editing and storage. I have to admit I do also use it occasionally to access various trusted photography related websites but I have never used it for email. A requirement of another (non-photography related) contract is email and Microsoft Office, both of which I have installed on the laptop.

Bottom line - I think I've got myself into a bit of a complicated mess with multiple machines and backup devices - and not sure of the best way to streamline everything without shelling out too much money

 

[Marsu42]

In any event though, I've always tried to keep the Mac Pro as 'clean' as possible by using it exclusively for image editing and storage.

... again, this is exactly what virtual machines are for.

You get a nice, clean main environment and simply stuff all other applications in vm boxes that just are one large, portable file on your hd. Given the computing power of newer machines, except for image processing you can run most programs in a vm w/o a noticeable performance hit. Well, better learn late than never :->

Edit: Another hint: For just running standard apps like IE or Outlook, you don't even need a full vm and a windows license - look at Crossover: https://www.codeweavers.com/products/crossover-mac/

 

[aj1575]

I just have one question before I like to go into the hardware upgrade.

Did you analyze your data. What do you save (JPEG, RAW, Both), do you still need to keep all the files? These are questions I would ask myself before upgrading the hardware. Because the upgrade only works temporarly, until the next HD bigger HD is full.

 

[tomscott]

Quick fix - buy two 4tb drives, or two 5tb and remove one 2tb so you populate the 4 bays. Have one 4TB as your main storage drive, use the second as a time machine back up. Put the other 2 2tb drives in a raid 0 and use that as back up number two internally.

If you need to, buy another 4tb and run weekly backups and keep it off site for safety.

Get a 256gb SSD and run your apps off that.

Thats the simple way of doing it.

Second is to buy a raid array which are expensive, either a 6 or 8 bay fill it with 2 3 or 4tb drives and have it constantly running. Don't underestimate the importance of an offsite back up, if you have a disaster such as flooding, roof leak, fire etc thats all your files gone even tho you have made two back ups it doesn't make you completely safe as its in the same place. You can buy fire and water proof arrays now, but again very expensive.

Depends how much you have to spend and how much you shoot. I keep telling people that good storage is a necessity although I get lots of stick for it. Don't underestimate it, and make sure you give yourself enough room to expand in a 3 year period then you don't have to worry about a huge cost upfront then again in a few years time. Adding drives as you need isn't a good strategy IMO. It is expensive to do it correctly, especially if you are shooting important events its up to you to ensure they are not lost.

Another issue is when you upgrade cameras. My 5DMKIII produces 23mb Raw files, edit one and the PSD will come out about 10 times the size around 250mb. Shoot a wedding or event at 1500 images which is around 35gb, store those incase they come back and want more. Edit 300 of them at 250mb so 75gbs, total of 110gbs. Do this once a week you will rack up 5.3tbs per year. We all know as photographers you can't live off one event a week so it quickly racks up. Then if you upgrade to a higher MP camera aka the rumoured 50mp camera double that again! So worth baring in mind, many people really underestimate their storage needs, future proofing isn't a bad idea.

For me physical is better than cloud as in the UK where I am the internet is not fast enough to move the size of files I produce. Cloud storage can be cheap from £20 a month for 1tb.

I have a mac pro I have the main 4 bays filled with 4tb drives, and have one 256gb SSD running off the PCI express port. I also have a 2tb drive in the spare DVD drive slot running from a spare Sata 3 on the motherboard. So 6 drives in total, I then have an external 16tb raid array and another that is offsite.

 

[ereka]

I just have one question before I like to go into the hardware upgrade.

Did you analyze your data. What do you save (JPEG, RAW, Both), do you still need to keep all the files? These are questions I would ask myself before upgrading the hardware. Because the upgrade only works temporarly, until the next HD bigger HD is full.

You make a very good point!

For the car and real estate contracts, I shoot exclusively RAW, process in LR & PS, then batch export JPEG as required. I could delete the jpegs after uploading to the clients as I could easily re-export as and if needed. I'm contractually obliged to keep copies of all images for at least 6 months, but after that I could in principle also delete the RAW files.

For my personal work however and some social photography (e.g. beauty retouches) I end up with TIFF files in addition to the RAW files. I'd not want to delete the TIFF files as they incorporate a lot of time consuming retouching work or the RAW files as they incorporate the original data and would be needed if I decide to have another go at the retouching at a later date. Again though, you are quite correct that I could delete the jpegs at some point rather than store them indefinitely. There's also a lot of really old stuff, particularly on the LaCie external disc, that I hardly ever look at and I doubt anyone else would be interested in now but I have a natural reluctance to delete anything and it is sometimes interesting to look back and see how my photography has progressed over the years (it's humbling to look at old work that I was happy with at the time but to my eyes now looks very poor indeed).

So, yes - a little housekeeping would undoubtedly hold the fort for a short while. Even given my inherent reluctance to delete anything, I think increasing my storage capacity to 4Tb combined with ongoing good housekeeping as you've suggested would keep me going probably for several years.

 

[Marsu42]

What do you save (JPEG, RAW, Both), do you still need to keep all the files?

Expanding on that idea: If they're raw files, converting archived shots to lossy dng might be good idea (w/o jpeg preview since LR/PS don't use these anyway). It about halves the size of the files with nearly no loss of iq and, unlike jpeg, you keep the ability for lossless wb adjustment.

 

[aj1575]

[quote author=ereka]
So, yes - a little housekeeping would undoubtedly hold the fort for a short while. Even given my inherent reluctance to delete anything, I think increasing my storage capacity to 4Tb combined with ongoing good housekeeping as you've suggested would keep me going probably for several years.
[/quote]

Make yourself a plan of what you need to keep, and what you like to keep. For costumer work, I would keep all the files as long as needed plus a few month (they want 6 month, keep em 9 or something), but after that you can actually get rid of all the stuff. If you like to keep something of that work, I would burn the JPEG on DVD. The possibility that you need RAW files of these shoots is very small, but if a costumer comes back for whatever reason, you still have something to sell/give him.

For your own work, I would be very drastic in cutting down the number of keepers; let's be honest, nobody looks at 20'000 pictures of something. So keeping down the numbers, helps the quality of your collection, saves diskspace, and is a good exersice in rating pictures. (This process is a more timeconsuming than buying new hardware, but in the long run it only helps you).

 

[Skirball]

I don't know how much retouch you do, but I suspect that the edited files have a lot to do with your issue. I'm regularly in the 2 - 4 gB range with my working files, both for retouch and for architectural work. Files this size eat up space quickly, and there's no way I'm deleting them. While I think it's a good idea to scour through your folders and delete old stuff and lousy photos you're never going to use anyway, I find that I actually gain very little from it - other than a cleaner file system. JPGs are nothing - I could spend hours deleting old files and hardly make a noticeable impact. Even RAW files seem small compared to TIFF/PSD/PSB. I actually find I free more space going into old layered files and cleaning those up, deleting superfluous layers or masks, merging layers, and whatnot. I'm now better about doing that during my editing, but I never used to think about it.

As to storage, have you thought about just using external drives? There's some very reasonably priced options with USB 3. And the nice thing is you can physically separate it from your other files, for a much more secure backup.

 

[ereka]

I don't know how much retouch you do, but I suspect that the edited files have a lot to do with your issue. I'm regularly in the 2 - 4 gB range with my working files, both for retouch and for architectural work. Files this size eat up space quickly, and there's no way I'm deleting them. While I think it's a good idea to scour through your folders and delete old stuff and lousy photos you're never going to use anyway, I find that I actually gain very little from it - other than a cleaner file system. JPGs are nothing - I could spend hours deleting old files and hardly make a noticeable impact. Even RAW files seem small compared to TIFF/PSD/PSB. I actually find I free more space going into old layered files and cleaning those up, deleting superfluous layers or masks, merging layers, and whatnot. I'm now better about doing that during my editing, but I never used to think about it.

As to storage, have you thought about just using external drives? There's some very reasonably priced options with USB 3. And the nice thing is you can physically separate it from your other files, for a much more secure backup.

Good point - even a flattened TIFF is typically around 250 Mb whereas the jpegs I send off to my clients are generally only between 1 Mb and max 20Mb. I don't do a huge amount of retouching but, like you, having spent many hours working on the files there's no way I'm going to delete them! Definitely external drives for backup - I need to fix that but need recommendations on the best solution. Also, how to avoid having to use Dropbox to share files between Mac and PC.

 

[Don Haines]

Keep an off-site backup copy of your data. If something happens at your house you loose everything.