Bigger Pixel Camera Storage Requirements: What is your setup/plans for upgrades?

cayenne

EOS 5D Mark IV
Mar 28, 2012
2,307
312
Hi all,

Well, with the R5 coming out along with other new cameras with larger "electronic MF" format lenses, storage requirements are going to increase just for the simple RAW files themselves, but also for intermediaries when roundtripping from PS/AP or other editors for heavy lifting, etc.

What do you have set up now?

How do you plan to expand?

My working system for awhile now, has been to keep my application data on the computers HD...but I keep all image files external.

I have a 3TB working drive that I import my images and videos to, and do most of my work there, and keep a good bit of backlog there until I eventually age some of it off to my "backup" NAS.

(I'm using "backup" in quotes as that I know if not offsite it isn't truly backup, but I"l come to that later).

I also have a SSD drive external that I used for my application cache needs...these are older and on thunderbolt 2, but still seems plenty fast enough for what I'm doing.

I have an older 2 disk NAS that has 2x 3TB drives mirrored...and I do regular "backups" to that from my desktop external 3TB work drive.

I have a 4x disk NAS that has a total of about 8TB in Synology's RAID 5 equivalent....on here when I import images to my work drive I do "backup" import to here.

ON this NAS I also keep copies of most of my applications' install binaries, and assorted other assets (video music, effects, etc).

This still works and decent room left working with the 5D3 content.

But with the estimated 45MB sensor of the R5....well, files for stills are gonna get larger. If this thing is pumping out RAW video, well, that's gonna be a large space sink too.

So, what do you currently have?

What do you have planned in the future for your work environment and archiving/ backup needs?

To address true backups....offsite, do any of ya'll do this?
I've been looking at BackBlaze as a service to maybe use?

What do you use or plan to use for true backup offsite?

Thanks in advance,

cayenne
 

pj1974

80D, M5, 7D, & lots of glass and accessories!
Oct 18, 2011
666
168
Adelaide, Australia
Hi cayenne

Thanks for your post - you raise some good points and questions. Everyone's situation is different, so I'll respond briefly with my current set up - and what I hope to do in time when, I aim to get the R5.

Currently I have multiple Canon DSLR and mirrorless bodies (mainly APS-C). The Canon 80D and M5 are 24 megapixels (MP), so yes, the 45 MP of the R5 is going to produce larger files, I expect about double the size. (I shoot in CR2 / RAW 95% of the time).

My current set up in terms of computer, workflow and storage is that when I copy the photos from the cameras to my PC (for each day/occasion/event), is that I go through the images, choose the 'best' of the best, and put in a 'favourite folder' for that day. I view in ACDSee and batch-process in DxO Optics Pro, and also use Affinity Photo. I backup my images (both RAW and JPEGS) onto multiple external HDDs - and have at least 1 version/copy of these images (all formats) off site in my secure cupboard / filing cabinet at work.

I keep enough free room on my PC's built in SDD for the fastest operation, but I also have a normal internal disk HDD on my PC for immediate storage there (which I then copy from PC to external HDD, to have extra copies asap that way).

When I get the R5 (which may not be 'immediately upon release' - mind you) I plan to upgrade my software (i.e. when ACDSee will be able to view the CR3 files) and DxO Optics Pro / Affinity. I know my computer will be able to handle the files, it zips through the 24MP files fast. And I have processed 50MP files (from 5DS) on my computer without issue - it was quite speedy.

Maybe I will double my RAM if I feel it requires it, but I think it'll be fine. I will probably get a 2nd internal HDD (like a 4TB one) and keep backing up images on my multiple (many!) external HDDs. That works best for me, and I keep in control of things, where copies are.

By the way, I use a cable to copy / transfer images from my cameras to my PC most of the time. (I do this so I don't need to eject / re-insert cards). So you can imagine, I'm looking forward to a camera with USB3 speed! My mobile phone transfers images of similar size (in MB) to my PC notably quicker than any of my Canon DSLRs.

Well, that's my answer! Regards

PJ
 

koenkooi

EOS R
Feb 25, 2015
1,180
968
[..]
Currently I have multiple Canon DSLR and mirrorless bodies (mainly APS-C). The Canon 80D and M5 are 24 megapixels (MP), so yes, the 45 MP of the R5 is going to produce larger files, I expect about double the size. (I shoot in CR2 / RAW 95% of the time).
[..]
The lossless compression in CR3 is pretty good, so a 45MP CR3 will be less than twice the size as a 22.5MP CR2. A quick check of my library shows that at ISO 100 an 18MP CR2 (EOS M) is 22MiB, a 32MP CR3 (EOS M6II) is 32MiB. So CR3 is about 20% smaller. So instead of 55MiB per photo it would be 45MiB per photo.
 
  • Like
Reactions: pj1974

cayenne

EOS 5D Mark IV
Mar 28, 2012
2,307
312
The lossless compression in CR3 is pretty good, so a 45MP CR3 will be less than twice the size as a 22.5MP CR2. A quick check of my library shows that at ISO 100 an 18MP CR2 (EOS M) is 22MiB, a 32MP CR3 (EOS M6II) is 32MiB. So CR3 is about 20% smaller. So instead of 55MiB per photo it would be 45MiB per photo.

But, do you think most people will be doing the full RAW and not going compression? True RAW is RAW...right?

C
 

koenkooi

EOS R
Feb 25, 2015
1,180
968
But, do you think most people will be doing the full RAW and not going compression? True RAW is RAW...right?
You can't turn off the lossless compression on the current crop of CR3 cameras. It's true RAW, just stored more efficiently. Don't confuse it with 'C-RAW' with will compress lossy first and then lossless. It will still give you the same DR and features like non-destructive white-balance adjustments, but it removes some data where Canon thinks you won't notice it.
From the testing I did I couldn't spot the difference between the same shot with RAW and C-RAW, the consensus on the internet seems to be the same. I still shoot in regular RAW, since I don't want to run into possible C-RAW limitations unexpectedly in the future. But that's a combination of my snobbery and OCD, not supported by actual data :)
 
Last edited:

cayenne

EOS 5D Mark IV
Mar 28, 2012
2,307
312
You can't turn off the lossless compression on the current crop of CR3 cameras. It's true RAW, just stored more efficiently. Don't confuse is with 'C-RAW' with will compress lossy first and then lossless. It will still give you the same DR and features like non-destructive white-balance adjustments, but it removes some data where Canon thinks you won't notice it.
From the testing I did I couldn't spot the difference between the same shot with RAW and C-RAW, the consensus on the internet seems to be the same. I still shoot in regular RAW, since I don't want to run into possible C-RAW limitations unexpectedly in the future. But that's a combination of my snobbery and OCD, not supported by actual data :)
Ahh, OK...I didn't realize that the new RAW would by it's nature be compressed at all....I love learning new things!!
:)

But even so, with sensor size increasing and again, not just for stills, but I have to imaging that video will be chewing up a lot of space, as that I understand you'll be able to shoot RAW video 8K with the R5?

So, with this combined and the ever increasing file sizes on the horizon, it is interesting to hear what others are saying they are planning to increase their work and storage capacities....

C
 

koenkooi

EOS R
Feb 25, 2015
1,180
968
[..]
But even so, with sensor size increasing and again, not just for stills, but I have to imaging that video will be chewing up a lot of space, as that I understand you'll be able to shoot RAW video 8K with the R5?
[..]
RAW video on the 1Dx3 is limited to 12-bit (all e-shutter modes are), I expect the R5 RAW video to be the same. There's also funny stuff going on, since the bitrates mentioned by Canon are far, far lower than you would expect for 5.5k RAW.

But yes, ginormous files for RAW video. In his review of the Sigma FP, Caleb from DSLRvideoshooter mentioned that a trick to conserve space if to use e.g. Davince Resolve to convert the files to ProresRAW or BMRAW to save space while retaining editing flexibility. I don't do video, so I have to take his word for it, but it sounds like a good trade-off.
 

LDS

EOS 5D Mark IV
Sep 14, 2012
1,671
215
You can have 8TB NAS disks at about $200-250 and 6TB ones at $150. 14-16TB models are available if you need. If your actual disks have some years of usage, it may be time to update them while buying larger ones. You need to ensure your NAS and the like can support the larger disks.

You can easily find how many photos you shoot per year, and calculate the space required by larger files and what space you'll need for the next n years.

If you do video your storage needs will be much higher. You'll have to decide in which format keep them.
 
  • Like
Reactions: cayenne

Random Orbits

EOS 5D Mark IV
Mar 14, 2012
2,386
260
I currently use a synology NAS. 4x2TB, and that has served me well for the last few years. All my video and photo files are on the NAS, and my computer backs up to the NAS as well. I back up the photos/video files and other documents about once a quarter to separate external hard drives. The computer, NAS, and external hard drives are all in the house (so technically not "off-site"), but they're located in different places in the house). I plan on getting a safe/gun safe for storing the external hard drives for fire protection.

I've been thinking about replacing the NAS for a while, and I'll probably look to build one in the next 12 months or so. I just built a new computer last December, so the editing side is all set. I'll have to look at FreeNAS, it has options that use more parity drives, so it can suffer multiple drive failures before having a catastrophic failure. Plus, I've heard it's easier to port the data drives to another machine when you want to upgrade/replace the host hardware. For up to a 4 disk NAS, I'd probably stick with synology, so the FreeNAS/build option is for a NAS with more physical disks.
 

koenkooi

EOS R
Feb 25, 2015
1,180
968
I currently use a synology NAS. 4x2TB, and that has served me well for the last few years. All my video and photo files are on the NAS, and my computer backs up to the NAS as well. I back up the photos/video files and other documents about once a quarter to separate external hard drives. The computer, NAS, and external hard drives are all in the house (so technically not "off-site"), but they're located in different places in the house). I plan on getting a safe/gun safe for storing the external hard drives for fire protection.
[..]
Please pay attention to the kind of fireproofing the safe has, having the inside at 100% humidity isn’t going to favor electronics.
 
  • Like
Reactions: Random Orbits

cayenne

EOS 5D Mark IV
Mar 28, 2012
2,307
312
I currently use a synology NAS. 4x2TB, and that has served me well for the last few years. All my video and photo files are on the NAS, and my computer backs up to the NAS as well. I back up the photos/video files and other documents about once a quarter to separate external hard drives. The computer, NAS, and external hard drives are all in the house (so technically not "off-site"), but they're located in different places in the house). I plan on getting a safe/gun safe for storing the external hard drives for fire protection.

I've been thinking about replacing the NAS for a while, and I'll probably look to build one in the next 12 months or so. I just built a new computer last December, so the editing side is all set. I'll have to look at FreeNAS, it has options that use more parity drives, so it can suffer multiple drive failures before having a catastrophic failure. Plus, I've heard it's easier to port the data drives to another machine when you want to upgrade/replace the host hardware. For up to a 4 disk NAS, I'd probably stick with synology, so the FreeNAS/build option is for a NAS with more physical disks.
I'd looked at trying to build a dedicated FreeNAS system a few years ago, I bought drives, etc.....and I just ran into it being too much a PITA vs getting stuff working and done quickly.....and I've had decent experience working with Linux/Unix type systems....and I gave up on that idea and bought commercial.

Now, I'm guessing things have improved a bit since then....so, I'd be VERY interested in hearing your progress on that for sure!!

I've been happy with the Synology stuff and I may buy a larger (>4 drives) one soon for greater onsite BU type storage...and then look to maybe change my external working drive(s)....I got a Mac Pro not long back and thinking the thunderbolt 3 stuff or usb-c connections would be better than what I have now that is older SATA drives on a thunderbolt 2 caddy....

Not that I feel I'm running into a problem yet, but thinking to future proof myself, especially if I start to dabble in 8K RAW video....

C
 

Random Orbits

EOS 5D Mark IV
Mar 14, 2012
2,386
260
I'd looked at trying to build a dedicated FreeNAS system a few years ago, I bought drives, etc.....and I just ran into it being too much a PITA vs getting stuff working and done quickly.....and I've had decent experience working with Linux/Unix type systems....and I gave up on that idea and bought commercial.

Now, I'm guessing things have improved a bit since then....so, I'd be VERY interested in hearing your progress on that for sure!!

I've been happy with the Synology stuff and I may buy a larger (>4 drives) one soon for greater onsite BU type storage...and then look to maybe change my external working drive(s)....I got a Mac Pro not long back and thinking the thunderbolt 3 stuff or usb-c connections would be better than what I have now that is older SATA drives on a thunderbolt 2 caddy....

Not that I feel I'm running into a problem yet, but thinking to future proof myself, especially if I start to dabble in 8K RAW video....

C
Yes, we shall see what happens by next year, lol! I wouldn't try to build a NAS for 4 bays... for 6 or 8 or more, then it might be worth figuring out.
 

LDS

EOS 5D Mark IV
Sep 14, 2012
1,671
215
I'll have to look at FreeNAS, it has options that use more parity drives, so it can suffer multiple drive failures before having a catastrophic failure. Plus, I've heard it's easier to port the data drives to another machine when you want to upgrade/replace the host hardware.
FreeNAS is built around the ZFS file system. You can increase redundancy with RAID too (i.e. RAID-6, is supported by the controller), the difference is that ZFS is managed by the operating system, while RAID is managed usually by the hardware controller. While you can use software RAID as well, ZFS is designed from the start to be a software solution that takes advantages of working at an higher level and with more resources. That's why you can aslo move disks easily to another machine as long as it supports the same ZFS version (or newer).
 

Mt Spokane Photography

I post too Much on Here!!
Mar 25, 2011
16,114
1,204
I currently have a SSD working disk, and 3TB storage disk with a couple 12 TB NAS. I've been waiting for a good deal on larger disks, otherwise, I'd probably pop 8 TB disks in my Nas and a 4TB SSD in my pc. I don't see a 45 MB sensor with 60 MB files as a big deal, storage cost is extremely cheap.
 
  • Like
Reactions: pj1974

SteveC

R5
Sep 3, 2019
1,205
972
FreeNAS is built around the ZFS file system. You can increase redundancy with RAID too (i.e. RAID-6, is supported by the controller), the difference is that ZFS is managed by the operating system, while RAID is managed usually by the hardware controller. While you can use software RAID as well, ZFS is designed from the start to be a software solution that takes advantages of working at an higher level and with more resources. That's why you can aslo move disks easily to another machine as long as it supports the same ZFS version (or newer).
Hardware is a bad idea.

If the board goes bad you lose absolutely everything, and it has happened to a friend of mine who was running mirrored drives. He was safe from one drive failing, but not safe from the board smoking.
 
  • Like
Reactions: pj1974

LDS

EOS 5D Mark IV
Sep 14, 2012
1,671
215
If the board goes bad you lose absolutely everything, and it has happened to a friend of mine who was running mirrored drives. He was safe from one drive failing, but not safe from the board smoking.
It's a choice when you have a maintenance contract and will be supplied identical hardware (or at least compatible). Usually not a great idea for consumer one. Hope your friend had backups - which should be made exactly for this kind of situations.
 

SteveC

R5
Sep 3, 2019
1,205
972
It's a choice when you have a maintenance contract and will be supplied identical hardware (or at least compatible). Usually not a great idea for consumer one. Hope your friend had backups - which should be made exactly for this kind of situations.
I don't know if he did or not.

I do know that a *software* mirror isn't considered a backup either, though it is for the simple case of a drive failure. It won't do anything to guard against your accidentally deleting your data, which is why they tell you it's not a backup.

Here's what I personally did (I am assuming some knowledge of ZFS, and the difference between a "storage provider" (single disk) vdev (group of disks raided together to form a virtual disk) and a zpool (a striped concatenation of one or more vdevs).

I'm running a FreeNAS mini XL, I have six 4TB drives on it set up as a RAID-Z2 (their variation of RAID 6) vdev and zpool, with one of the six drives being a hot spare, so basically 3 disks worth of storage with double redundancy. You can lose two drives off of that at the same time. (They also have Raid Z3, where you can lose up to three drives.)

My main gripe with the ZFS system, however, is that you cannot simply add more drives to an already existing vdev, like you could on a synology. If you add a disk to that zpool, It will instead treat that new disk as its own vdev, and stripe your data between it an the other vdev, basically creating something like a Raid 0...and if that disk you just added ever fails, you're toast. (You have to plan ahead, and use several new drives to create another raid-esque vdev, THEN stripe them--you at least cannot be burned unless one or the other of the two vdevs fail and they each have their own redundancy.) If you want to increase the capacity of an already existing volume, you can buy larger hard drives and swap them out one by one and let the system "resilver" them.

The other two drive bays on my system each have a larger hard drive 6TB, as stand alone vdevs and pools, and I use them both as backups. I also copy everything onto a 4TB passport which is also its own pool--I have three of them and I rotate them as off site backups. This works now because at the moment I only have 2TB or so of data even though my "main" pool is good for about 12 TB total.

With hard drives getting to be so capacious, by the way, the ZFS people recommend at least a Raid-Z2 setup and even that makes them a bit nervous (they fear having additional drive failures while you're recovering from the first one), and they'd prefer you use a Raid-Z3.
 
  • Like
Reactions: cayenne

LDS

EOS 5D Mark IV
Sep 14, 2012
1,671
215
It will instead treat that new disk as its own vdev, and stripe your data between it an the other vdev, basically creating something like a Raid 0...and if that disk you just added ever fails, you're toast.
Yes, you should never add a single disk - you'll need to add another whole RAIDZn vdev.

the ZFS people recommend at least a Raid-Z2 setup and even that makes them a bit nervous (they fear having additional drive failures while you're recovering from the first one), and they'd prefer you use a Raid-Z3.
IMHO that is important when you need to ensure no downtime of your NAS. Instead of being paranoid about a single NAS fault tolerance, I prefer to replicate snapshots to another storage. In the rare case a whole system fails. I can rebuild if from the copy. It means "downtime", but in my case is fully acceptable.

What I don't like about FreeNAS (soon to become TrueNAS CORE) is their "all or nothing approach", for example they discourage approaches which are rightly bad in an enterprise environment, but not so much in a small, single user ones, like replicating a snapshot to external disks via USB - even including the much safer and more performant USB 3. For a single user such copies even without the full reliability of a secondary system is better than nothing.
 

SteveC

R5
Sep 3, 2019
1,205
972
Yes, you should never add a single disk - you'll need to add another whole RAIDZn vdev.



IMHO that is important when you need to ensure no downtime of your NAS. Instead of being paranoid about a single NAS fault tolerance, I prefer to replicate snapshots to another storage. In the rare case a whole system fails. I can rebuild if from the copy. It means "downtime", but in my case is fully acceptable.

What I don't like about FreeNAS (soon to become TrueNAS CORE) is their "all or nothing approach", for example they discourage approaches which are rightly bad in an enterprise environment, but not so much in a small, single user ones, like replicating a snapshot to external disks via USB - even including the much safer and more performant USB 3. For a single user such copies even without the full reliability of a secondary system is better than nothing.
Yes, you do run into the purists. On the IxSystems forum, some people's hair will ignite at the suggestion of doing offline backups with USB drives, but others are a bit more realistic. (I have no choice; my network connectivity will not allow me to put my 1.5 or so terabytes of data offsite over a network connection.) One thing, though that they WILL insist on is that you really ought to format that USB drive with ZFS as well. Don't even think of trying to format it as NFS or ext4 or (eeep) NTFS. But that means you'd better have a machine set up to read it, in case your NAS goes completely belly up.