Canon 5D3 SD and CF testing (slow slot problem)

[chromatropic]

There's general dismay over the crappy SD performance of the 5D3. There are various theories I've read about the performance and what the camera is doing, one of them is that the camera is downclocking the CF card. Canon wouldn't comment on it in their forums other than to say they don't comment on third party websites.

So I did some testing today with three cards:

Lexar 800x 128G CF = ~50MB/sec write speed (rated write speed 120MB/sec)
SanDisk Ultra 30M/s 64G SDXC = ~10MB/sec write speed (sigh) (rated write speed 30MB/sec)
SanDisk Extreme Plus 64G Micro SDXC = ~20MB/sec write speed (rated read speed 80MB/sec, this hits the camera's hardware cap for writing to SD)

I wanted to find out if the camera was indeed downclocking the CF card and/or what the algorithm was in terms of writing to the cards. I came up with four propositions:

1. The cards were written to serially at their fastest rate.
2. The cards were written to in parallel at their fastest rate.
3. The faster card was downgraded to the speed of the slower card with serial writes.
4. The faster card was downgraded to the speed of the slower card with parallel writes.

I considered that one or several of these approaches might be used, depending on the configuration of the camera in terms of RAW and JPEG and which card was receiving which.

For my method I just focused on the same subject and shot away at high speed until the buffer was full, then let the buffer drain. I marked time at the buffer fill point and when the writing stopped. There were two exceptions in that the faster SD card paired with the CF, in L,L mode, I couldn't fill the buffer, so I stopped after 25 seconds and then let it drain. And, with the faster SD card two of the algorithms looked like they could predict the result and I wanted to try to differentiate them so I continued to run the shutter past the buffer fill point. The result did not change much. Cards were formatted (SD to low level) between every test run.

I used each card running solo in RAW mode to obtain a baseline reading/estimate for its write speed figuring this would have the least camera processing overhead and the bulk of the work is writing to the card.

These speeds and the data written during each test run were then used to plug into the 4 theories above, and the theories who's prediction matched the observations were flagged out with green boxes on their predicted times.

Some basic observations:

1. writing to multiple was the same as writing separate if the format was identical, so this didn't have any benefit or detriment.

2. no downclocking prediction better explained something that simple serial and parallel predictions could explain, and the simple parallel and serial predictions explained some configurations better than the downclocking predictions. So it would seem there is no downclocking and this I think was just a handwaving explanation that people have come up with to explain the degraded performance when an SD card is plugged in, especially if RAW was being used.

3. Canon sucks for crippling the 5D3 SD slot. It's crippled though, not useless as can be seen below.

4. In the table the MB/s rates are calculated for the amount of data written to a card and then using the total time for the camera to finish writing. So it's an "effective" number that shows the gimping down based on the SD card from the max rate of the card. The T() columns indicate the time predicted to handle the writing load in that configuration for the card based on the observed speed for the card operating in solo, RAW mode (least overhead for the camera).

5. The effective performance of the various configurations is in the eFPS column. This is the number of frames you are getting per second with the buffer discounted (continuous shooting). So, the fastest performance (I didn't test simple L mode on any cards) in the chart is L,L with a 20MB/s write SD card. This is faster than writing RAW solo to an 50MB/sec CF card. So if you are not a RAW shooter then you can do just fine using the SD card as a failsafe as-you-go backup. In this case I couldn't fill the buffer.

6. The absolute no-no is writing JPG to CF and RAW to SD. This brings you down to a best case performance of 0.3 frames per second which is only 20% of the speed of the CF card on its own writing RAW.

Conclusions

1. Downclocking theory holds no water. It would appear that the best theory for the 5D3's algorithm is simply that it is writing to both cards in parallel to the max speed of the card when it has matching formats (R,R) or (L,L) and I would assume other similar configurations, and so just ends up capped by the gimped SD card performance. When formats mismatch, the camera cannot write in parallel, but must write serially. I'd assume first writing the RAW image, then doing the JPG conversion, and writing the JPG.

2. knowing how the camera deals with the SD card does give you some options to use the SD card still... An SD card can cause exaggerated damage to perceived performance then because the write speed of the slot is capped, plus as in my case, a card labelled "30M/s" only has "10M/s" write speed, and then the use of a mismatched format will trigger serial writes... and if the larger data format is written to the SD card you get a massive disaster for performance. if you use it to create backup JPG files at S1 size then you should get maybe 85% of the speed of shooting CF RAW at full size itself. This is not bad. CF/MRAW + SD/M in my test ran at 98% of the speed of full sized RAW on its own. CF/mRAW + SD/L was 84% of full sized RAW on its own. I think these are all viable configurations, I like mRAW for my shooting and I like to have the L sized JPG just in case I needed more resolution.

3. some configurations are performance killers, RAW should never be written to the SD, ever ever basically. Never using an SD card in any configuration though is a paranoid solution based on the downclocking theory which I think these tests discredit.

4. using an SD card incapable of achieving 20MB/sec is going to harm you, and from what I've read this is the fastest the camera can do on the SD slot. It looks like the rule of thumb may be to divide the read speed by three to guess at what it will do in the Canon 5D3 and cap it at 20MB/sec. Remember to never judge the write speed of these cards by their labels...

6. The microSD in an adapter seems to be a perfectly fine option. I got this to use in a GoPro Hero but will now share it with the 5D3. However, it formatted a lot slower than the 64 gig Ultra card.

7. I'd pay to fix this SD slot. My preference was always to use SD for convenience because I can just slip it into my Macbook without needing a card reader, but this is the end of that.


You are free to use / modify / publish / criticize / improve / disprove these results here or on any forum, blog, website, whatever. In my reading over the last couple of days I didn't see anyone trying to comprehensively test these configs to try to answer the questions, just a lot of hearsay and guesswork. So if you use this for anything just credit/blame Chromatropic please.

 

[neuroanatomist]

Thanks for putting in the effort!

Conclusions, cont'd:

8. I'm glad my 1D X has dual CF slots, and not CF+SD.

 

[mackguyver]

Thanks for putting in the effort!

+1 and welcome to CR.

A couple of notes of agreement on your tests -

1. The cards don't have the same write speeds as read speeds, so your Lexar 800x does not write at a rated 120 MB/s. If you look at Lexar's site Lexar's site, it says, "*Up to 120MB/s read transfer, write speeds lower." . The same applies to your Sandisk cards. This is why the 800x cards are considerably less expensive than the 1000x and 1066x cards. The read speed isn't much lower, but the difference in write speed (50-60MB/s vs. 95 or 150 MB/s) is huge. I find the advertising pretty misleading, but it is what it is now that both of the big guys are doing this in their ads.

2. I believe the SD card slot of the 5DIII is limited to 30MB/s based on the older standard that was used when the camera was designed. A newer standard (UHS-I) was available when it launched, but probably not when it was designed.

3. I'd have to look for the source, but I'm pretty sure I've read that the use of both the CompactFlash and SD card slot (i.e. writing each shot to both cards every time you push the shutter) bumps the max write speed of the CompactFlash card to that of the SD card. It must use parallel processing that has to be kept in sync or something.

 

[chromatropic]

3. I'd have to look for the source, but I'm pretty sure I've read that the use of both the CompactFlash and SD card slot (i.e. writing each shot to both cards every time you push the shutter) bumps the max write speed of the CompactFlash card to that of the SD card. It must use parallel processing that has to be kept in sync or something.

Actually, the point of the testing was to try to verify that theory and it didn't do it. The predictions don't match the observations so I don't think that's what's happening. It fits in the simplistic cases but you can see that all four theories fit in the simplistic cases. It's like the blind men describing an elephant, one says it's like a thick rope, one says it's like a tree trunk and one says it's like a wall made of leather. If you stop at just that one case the theory does predict the description of an elephant very well. So you need to keep plugging in different scenarios to see if it holds up in every case.

This model is shown in the DCPP column. And in one case it predicts a write time of around 20 seconds for the data that's on the card, but it took only around 10 seconds.

The case of the 5D3 and the 1Dx, you have the first with two slots and one processor. The second has two slots and two processors. So in the case of the 1Dx you have a full parallel image processing pipeline that starts from capture and terminates at a card, basically it's got two 5D3s inside that body. If the two destinations have selected different transformations of that image, it's very simple because each of those processors can proceed independently. You will still be capped by the slower of the two transformations before the buffer can advance to the next capture to be processed.

In the case of the 5D3 you have a mess, one processor, with two different types of cards. If you have two different transformations of the captured data, that one processor must perform both tasks and that is by nature a serial task. So I put down one of the theories that the processor if asked for two different image transformations, would handle them serially. It makes some sense to think that this processor is not capable of housing two completely different image transformations at the same time. So it doesn't likely even have the data available to write in parallel in this case. So it does the only thing it can, does the first transformation from the captured data, writes it, and then starts again doing the second transformation, and writes that out to the second card.

The Digic5+ is a lot faster than the Digic 4, faster even than two Digic 4s, but this is not entirely a speed problem, it's a memory problem. Since the 1D4 and 1Dx have two image processing pipelines they can handle everything fully in parallel but this 5 is basically faking it with speed. The 5D2/Digic4 on its own would be too slow to manage two transformations so what you have with the 5D2 is just one slot.

The raw speed of the new processor means that it can perform two transformations in less time than the 5D2 took for one transformation. So this allows them to introduce the second slot in the 5. And with that comes the limitation of having to perform the operations in sequence if they are not identical. And with that limitation comes writing the data in sequence if the transformations are not identical because the host processor can't hold two transformations at the same time. I think

Anyway that theory's predictions all fit the data within an error rate of 10%. And that can easily be my error since I'm using a mechanical stopwatch with 0.5 second granularity and having to time by hand.

Occam's razor then applies, if one theory is more simple, all things being equal its more likely to be correct. But, it's also a much better fit for the data which I think is what seals the deal. I think in order to buy the slowed-down-card + writes in parallel it means you need to accept that the Digic5 can do two full transformations of the data in parallel and hold the results of those transformations and be able to write out all that data in parallel. I don't buy that, before it even gets to the point of it not matching the observed data (the camera is twice as fast this predicts in some cases). Basically, putting in a crappy SD card slows everything down... but it would really, REALLY slow it down if the CF card were matching the SD card byte for byte. So all the crappy SD card seems to be doing is just slowing down its own self, and the problem is made worse when you choose different transformations for the capture, because then you are no longer dealing with just the slower of the two cards being the maximum time, but you are adding the time for each of the two cards. But if the CF card stepped down too then in my test case above would have taken 20 seconds instead of 10.

I'd love to see if there is some official word on how they are accomplishing this. I couldn't find any which is why I just said screw it and tested it myself.

The above is just trying to explain the observations. There may be additional models that fit better than the four I listed. The CF card being stepped down to the absolute max write speed of the SD bus, if that is indeed 30MB/sec, would also be a close fit for the data but not as good as what the serial-on-mismatch, parallel-on-identical theory seems to fit (it misses in two places by about 22% and 40% where the serial/parallel theory misses on maximum by 10%).

The more cards and variations I test the more it would clear up because the correct theory will continue to produce correct results for all combinations.

10% may also factor in some additional overhead I missed, for instance after formatting the SD cards are showing more memory used (800k) than the CF card (about 160k). I spent enough time already before discovering that so didn't try to go and patch it back in.

 

[mackguyver]

I think in order to buy the slowed-down-card + writes in parallel it means you need to accept that the Digic5 can do two full transformations of the data in parallel and hold the results of those transformations and be able to write out all that data in parallel. I don't buy that, before it even gets to the point of it not matching the observed data (the camera is twice as fast this predicts in some cases). Basically, putting in a crappy SD card slows everything down... but it would really, REALLY slow it down if the CF card were matching the SD card byte for byte.

My thoughts, just barely educated guesses given the dearth of known information, are that the camera processes the data sequentially (reads then processes [if the DIGIC actually does this?] RAW data, creates JPEG), buffers this, and then writes in parallel.

In terms of testing, I can tell you that my Sandisk Extreme Pro (60MB/s write speed) cards write around 13-14 RAW files and my Lexar 1000x (90-95MB/s write speed) cards write around 17-18 RAW files before the buffer fills. I will have a new Sandisk Extreme Pro (150MB/s write speed) card on Wednesday next week and will see what kind of improvement that brings. Also, keep in mind that shutter speed, ISO, functions like ALO, CA removal, distortion correction, and the subject being shot all impact the file size and thus test results.

Apparently Magic Lantern has some utility to measure card speed and you might find some good information about this in their forum. I haven't tried ML but that might be a good place to look.

 

[mikejkay]

Chromatropic - great to get some data and many thanks. Is it possible/likely that Canon will produce a firmware update to "uncripple" the SD card slot? I already have UHS-1 SD cards, my bottleneck seems to be the older (legacy) 300x CF cards that I use. Anyway, many thanks for all your effort and for putting it into the public domain.

 

[Drizzt321]

Chromatropic - great to get some data and many thanks. Is it possible/likely that Canon will produce a firmware update to "uncripple" the SD card slot? I already have UHS-1 SD cards, my bottleneck seems to be the older (legacy) 300x CF cards that I use. Anyway, many thanks for all your effort and for putting it into the public domain.

This is likely a hardware thing. So it's not crippled, it's just the design at the time it was created. I'd be willing to bet that there's only 1 super-fast card 'port' on the DIGIC chip, and the other hangs off of essentially hangs off of the USB bus as a USB card reader, which basically limits it to 60 MB/sec theoretical max bandwidth, but because of the design age they only put in an SDHC (or was it SDXC?) which is the previous spec for SD cards which maxes out at ~30MB/sec theoretical performance.

As mackguyver put it, I'd bet that the camera pulls the RAW into memory, if it's supposed to write as RAW it generates the thumbnail JPG to write into the RAW file, and if it's supposed to write a JPG file it generates the JPG at the size & quality specified. At this point, it writes the file(s) out to the memory cards in parallel, and I'd bet via DMA (at least to the CF slot, which would help free up the CPU to an extent). At this point, if it finishes writing to the CF card, but the SD card is still waiting to finish it's write, it keeps that entire photo in memory (final output to both cards, so if JPG + JPG, the JPG for both, or if RAW + JPG, the RAW to the CF and JPG to the SD). So it essentially holds up a good chunk of your buffer while waiting to finish writing out to the SD slot. Once it's done writing that image, it clears out that particular images space in the buffer and you free up room for another photo in the buffer.

I think this is why, no matter what combination, continuous shooting for RAW+JPG is less than for just RAW. Both because of the extra CPU to generate the JPG, and the extra buffer space needed by the JPG.

And mikejkay, just go ahead and get a newer, higher speed CF card. You can get a quality Transcend 400x (90MB/sec read, 60 MB/sec write) UDMA7 compatible card for less than $40 from B&H. I've got them, use them quite a bit. Works great in my 5d3. It'll also just plain be faster than any SD card unless you regularly do a low-level format on the SD card as the UDMA7 supports TRIM, which lets the CF card clear out the blocks ahead of time instead of waiting until it actually needs to write to them.

 

[mackguyver]

Chromatropic - great to get some data and many thanks. Is it possible/likely that Canon will produce a firmware update to "uncripple" the SD card slot? I already have UHS-1 SD cards, my bottleneck seems to be the older (legacy) 300x CF cards that I use. Anyway, many thanks for all your effort and for putting it into the public domain.

This is likely a hardware thing. So it's not crippled, it's just the design at the time it was created. I'd be willing to bet that there's only 1 super-fast card 'port' on the DIGIC chip, and the other hangs off of essentially hangs off of the USB bus as a USB card reader, which basically limits it to 60 MB/sec theoretical max bandwidth, but because of the design age they only put in an SDHC (or was it SDXC?) which is the previous spec for SD cards which maxes out at ~30MB/sec theoretical performance.

As mackguyver put it, I'd bet that the camera pulls the RAW into memory, if it's supposed to write as RAW it generates the thumbnail JPG to write into the RAW file, and if it's supposed to write a JPG file it generates the JPG at the size & quality specified. At this point, it writes the file(s) out to the memory cards in parallel, and I'd bet via DMA (at least to the CF slot, which would help free up the CPU to an extent). At this point, if it finishes writing to the CF card, but the SD card is still waiting to finish it's write, it keeps that entire photo in memory (final output to both cards, so if JPG + JPG, the JPG for both, or if RAW + JPG, the RAW to the CF and JPG to the SD). So it essentially holds up a good chunk of your buffer while waiting to finish writing out to the SD slot. Once it's done writing that image, it clears out that particular images space in the buffer and you free up room for another photo in the buffer.

I think this is why, no matter what combination, continuous shooting for RAW+JPG is less than for just RAW. Both because of the extra CPU to generate the JPG, and the extra buffer space needed by the JPG.

And mikejkay, just go ahead and get a newer, higher speed CF card. You can get a quality Transcend 400x (90MB/sec read, 60 MB/sec write) UDMA7 compatible card for less than $40 from B&H. I've got them, use them quite a bit. Works great in my 5d3. It'll also just plain be faster than any SD card unless you regularly do a low-level format on the SD card as the UDMA7 supports TRIM, which lets the CF card clear out the blocks ahead of time instead of waiting until it actually needs to write to them.

I did some testing tonight and can confirm that the 5DIII is limited to 30MB/s - it's the same buffer depth as my 80MB/s card. 30MB/s is the max for Class 10 - UHS-I is the next standard, but the hardware likely wasn't available at scale when the 5DIII was designed. Fortunately the CF slot supports the latest UDMA7 150MB/s write speed cards!