DxOMark scores for 5DMkIII out - total score 81, 5DMkII had 79

Status
Not open for further replies.
Fishnose said:
Both are of course already working on coming generations - this is a long-term business.
I do genuinely wonder if this is really the case. I wouldn't be surprised at all if it turned out that development of the next 5D doesn't start for another year or two, given that it was 4 years between the Mark II and Mark III. There's clearly *not* 4 years of technological improvement in the Mark III, perhaps with the exception of the DIGIC processor, but this has been gradually developed and improved across the full range of Canon SLRs during this time period.
 
Upvote 0
My conspiracy theory:
I wonder if DxO is purposely picking the 8MP downsize option to boost Nikon scores. All nikon cameras that receive great scores have all had multiples of 8MP as the sensor output resolution. The 16MP D7000, 16MP D4, and 36MP D800. Since the DR calculation that Dxo uses is not based solely on the ratio of light to dark but only those values where a signal to noise ratio is below 0dB. I wonder if by perfectly dividing those numbers they are able to achieve a lower noise floor in the shadows enabling larger dynamic range numbers on paper. Notice the random Pentax K5 extremely awesome DR rating also contains a 16MP sensor. I also find it interesting that the $800 16MP Nikon D5100 has an equal DR to the Phase One...hmmmmmm. The Sony NEX-7 with its 24MP sensor (multiple of 8) and the 24MP D3x and 24MP Sony A580 also have top spots to both $40,000 medium format cameras known their dynamic range and all Canon bodies.

Can it really be a coincidence that EVERY top body in DR happens to have a MP count that is a multiple of the 8MP that DxO mysteriously uses for all of its calculations?
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Ratings/(type)/usecase_landscape
 
Upvote 0
I heard a rumor that Nikon is going to rebrand it's entire family of cameras as the "Amazon" series.

The D800 is reportedly going to be rebranded as the DxO-95.

Nikon has also reportedly updated their firmware to first display a side by side comparison of the 5D3 and (formerly) D800 DxO marks for ten seconds before reviewing the photo. Owners have apparently demanded this feature as a reassurance that their photo does indeed look good before actually reviewing the photo.

Also, the rebranded version of the new Amazon series are reported to now have red bodies with a black stripe instead of black bodies with a red strip. A Nikon spokesperson who would only speak on the condition of anonymity expressed that Nikon customers have historically been intimidated by the red stripe on competitors lenses and felt the red stripe on the (formerly) D800 was a good first step, but simply did not go far enough.
 
Upvote 0
Re: DxO results out for 5D3

sarangiman said:
Thanks jrista for your reply. A couple of things:

I guess I would call that making existing DR (in these cases, in the shadows) more usable, where as before downsampling it may be less usable. It can't increase DR beyond the capabilities of the hardware, however, regardless of how you work the math.

The DR the camera can capture/record is (partially) hard limited by the ADC, yes?

In other words, an 8-bit ADC would necessarily limit your DR to 8EV? Want to make sure I'm understanding this correctly. For example, if full-well capacity is 65,536e-, & you use a gain of 256e-/ADU, but now your blacks are crushed even if the read noise of your sensor is, say, 30e-. Even though your sensor may be differentiating blacks, a lot of blacks will be binned to an ADU of 1, correct?

If so, then, yes I totally see your point.

That would indeed be exactly correct. :)

Canon also uses a bias offset in each frame. The exact nature of it and exactly how it is used by a RAW processor is not fully known to me, but at the very least it sets a black point, allowing the total ADU range to span from -2048 to whatever maximumSaturation-2048 is. I don't know how that might affect results if you simply subtract 2048 from the original maximum saturation...to me, it sounds like you might be knocking off 2048 discrete levels of potential shadow luminance, since only the maximum saturation and stdev of noise is involved in DR calculations.

sarangiman said:
I'm not sure if a simpler strait forward equation like what LTRLI uses is indicative of real-world downsampling, either.

I wasn't talking about a downsampling equation... I don't remember LTRLI ever providing one. I was talking about his calculation of DR:
  • DR = log(base 2)[max pixel value in white file/read noise in ADU]

I understand that's a general way of determining PDR, but my issue, which I raised a number of times before, is that to relate this calculation to the DR the camera is capable of recording requires a number of assumptions that I do not know are valid! I even provided a thought experiment that, if implemented into the signal processing pipeline, would make Nikon appear to have greater dynamic range even though it may not (basically: if you bin certain low signals to 0, rather than allow both positive & negative variation around a black point of, like 2048, you can reduce the stdev of pixels in a resultant black frame).

Right, sorry. I know LTRLI's formula is not a downsampling formula, what I was trying to say is I don't think it takes into account the realities of downsampling properly. To put it another way, I think it produces a result without context, which is why it seems to produce much higher numbers for DR than would seem realistic when computing DR for a downsampled image. In its own right, taken at face value (without any context) its certainly not an invalid formula...but do the results produced by it have real-world applicability? Do they represent something real, or is the formula too simplistic? I used to believe that DxO's Print DR numbers, which are derived the same way, were simply demonstrating that once you eliminate noise, you are then realizing the sensors full potential. Kind of tough to keep believing that, though, when the D800's Print DR numbers are BETTER than the sensors full potential...its not quite like a star trek warp drive...you don't get 120% of maximum out of a 14-bit sensor. ;)

sarangiman said:
I don't see how you can really gain anything from the act of downsampling.

All I'm saying is: take a black frame from your camera, load it up in IRIS, measure the stdev of pixels. Now downsample that file 4x or something; now measure the stdev of pixels. I just did for a black frame from my 5DIII, & here are my results:

[list type=decimal]
[*]100%: stdev = 5.9ADU | DR=11.4EV
[*]25% (bilinear downsampling): stdev = 1.61ADU | DR=13.2EV
[/list]

Wow! ~2 stops more DR! Kinda makes you question that DR formula... what say you? :)

Wow! So odd... ;) I definitely question the DR formula...although it is interesting that you got 13.2 EV. I should have actually run that calculation myself to prove earlier points...that there is some biased skew between Canon Print DR results and Nikon Print DR results. Why is it that Nikon images gain so much more than Canon sensors when doing Print DR? Especially if Nikon images already contain a near-minimum noise floor and wouldn't logically seem to be able to benefit as much from normalization, where as Canon sensors still have some pattern noise forms at low ISO, and it would logically seem that they have MORE to gain, not less, from normalization.

Well, all I can say is yeah, the DR formula is fishy, and thats the key point I debate whenever these threads pop up. But I'm just a hater for saying so, and I personally don't care all that much about DxO results, so it doesn't really matter in the end anyway. ;P

sarangiman said:
(not even Sony cameras, which use the same Sony sensors as Nikon cameras do, seem incapable of keeping pace with Nikon cameras for some reason...which is VERY ironic.)

That is interesting & to me points to the signal processing pipeline in Nikon cameras doing something different that yields better results (real world? DXO? both?... I can't comment yet b/c I haven't yet done my 5DIII vs. D800 transmission wedge test, which I'll do as soon as my friend gets his D800).

Perhaps, but the major differences only seem to exist in normalized results. The two brands are a lot closer in performance when comparing native results...which makes me suspect that there may be some kind of software-level bias involved, rather than significant hardware-level differences. There is definitely a value to producing normalized results for consistency in comparison...but DxO is a bit of a black box. They expose some details and a little bit of math about their process, but there are also a lot of unknowns. Involving software in the process can affect results in a LOT of ways that could affect the objectivity of the final outcome. Who knows, for sure, exactly what their software actually does. Does it have any brand-specific optimizations that aren't publicized? If so, how are they skewing results?

The ironic thing here is that you can completely ignore the normalized results, and the native hardware results still indicate that Nikon and Sony sensors are technologically superior. Thats all someone really needs to say, and its not something that can really be (nor should be) disputed...but the Print DR results and funky S___ like "bonus points" are involved in the final "scores" that DxO gives to each camera. I find that to be fishy as well, and unnecessary. You don't need biased results to demonstrate that Sony Exmor based cameras offer superior IQ, and the actual meaningful differences between the best of the vest (D800) and the best of the competition (5D III or any one of the measured Digital MF cameras) are not really as big as DxO is making them out to be. Thats where my pet peeve lies...bias and skew when its unnecessary and not really honest or helpful to potential customers of any brand.
 
Upvote 0
skitron said:
The D800 is reportedly going to be rebranded as the DxO-95.

Nikon has also reportedly updated their firmware to first display a side by side comparison of the 5D3 and (formerly) D800 DxO marks for ten seconds before reviewing the photo. Owners have apparently demanded this feature as a reassurance that their photo does indeed look good before actually reviewing the photo.

Good stuff ;D ;D I heard that Nikon is also developing a 36-inch tablet that tethers wirelessly to the D800, I mean DxO 95. That way users can more easily see the astonishing detail in the snap shots of their dogs and fake floral arrangements.
 
Upvote 0
dswatson83 said:
My conspiracy theory:
I wonder if DxO is purposely picking the 8MP downsize option to boost Nikon scores. ... Notice the random Pentax K5 extremely awesome DR rating also contains a 16MP sensor. I also find it interesting that the $800 16MP Nikon D5100 has an equal DR to the Phase One...hmmmmmm. The Sony NEX-7 with its 24MP sensor (multiple of 8) and the 24MP D3x and 24MP Sony A580 also have top spots to both $40,000 medium format cameras known their dynamic range and all Canon bodies.

Remarkably, that 'all Canon bodies' includes the 16 MP 1D IV. Another conspiracy theory shattered against the cold, hard rocks of reality.
 
Upvote 0
Re: DxO results out for 5D3

Egad, sorry, but that's a whole lot of verbal diarrhea. As Inigo Montoya said, "Let me explain...no, there is too much, let me sum up..." (or analogize, as the case may be):

DxOMark scores are like the US EPA's automobile fuel economy figures. I'm not going to base a car purchase decision on that particular number. I've never come close to the mpg figures I'm supposed to be getting with any car I've ever owned. The number says nothing about how fun the car is to drive, how comfortable the seats are, or whether a dresser can fit in the trunk. Still, it's a useful number to use as one factor, among many more important ones, in a decision.

About normalization: "You keep using that word. I do not think it means what you think it means."

Will the debate ever end? "It'd take a miracle..."
 
Upvote 0
Re: DxO results out for 5D3

sarangiman said:
It's actually all of the other curves that are fishy . Ideally it WOULD be a straight line plot. It's read noise issues at certain stages of the read out that cause them to curve flat towards ISO100 for the non-exmor cameras.

Right, that's kind of what I meant... I stated it poorly. Now, don't you find it strange that the D4 also flattens at ISO 100? We're not supposed to believe it has worse read-out noise than the D800, are we?

Actually we are supposed to believe that the D4 has worse base ISO dynamic range than the D800 because it doesn't use any of the Exmor technology (and also more MP actually can tend to make it easier to get better read noise, but the main thing is the Exmore technology) so it was an expected result.

As for the method for calculating DR from white/black frames... I just don't see that as an accurate method, b/c of complications potentially arising from the signal processing pipeline.

It's not accurate at all as to what the sensor itself captures, the Canon sensors are grabbing way more DR than we measure. But it's totally reasonable because it's not like we can magically beam the direct sensor capture onto our cards, the camera has to use read out electronics to read the signal and convert it to digital bits.

If you maybe meant more because how it may play around with things, it seems that most things have been touched up in any sort of unfair for the vast majority of cameras. Once you get in exposures more than a few seconds I think Nikon does, unfortunately, play with the RAW files a bit. Some say the 5D3 may play with the RAW a bit at 25,600 and above, not sure. I think for a few months one of the Sonys was playing with the RAW files. Mostly when people check there doesn't seem to be an issue though.


For the basic dynamic range I think they just take their top blown out wedge (same as the white frame we talk about) and for then take a bottom wedge that contains the blackpoint (same as out lens cap on black frame) and then do the same thing we did

Yeah, I'm almost inclined to believe you, given that now my calculations this way from black/white frames of 5D2, 5D3, & D7000 suspiciously match DXO's numbers to almost a decimal point! Of course, if all these cameras religiously map white to that highest value on the white file & black to the mean of the pixels in the black file, with no other fancy signal processing in between, then maybe that's just a testament to DXOs testing protocol (if they actually base DR by imaging patches of a range of brightnesses).

FYI I started doing some DR tests on my 5DIII using a Stouffer Transmission Wedge (T4110). I'm working on my methodology here, but the following method yielded 11.2 stops for DR:

At ISO 100:
[list type=decimal]
[*]Find longest shutter speed at f/11 that is just short of no longer capturing any detail between Step 1 (not enumerated on wedge) & Step 2 (shooting RAW, using any/all sorts of HR & exposure reduction to pull detail out of highlights... i.e. +1/3stop from this shutter speed & Step 2 now has same signal as blown-out white file)
[*]Find shortest shutter speed where signal at Step 2 is ~12 (so, SNR of 2 if 5.9 is read noise, according to dark frame)
[*]DR = log2(longest shutter speed/shortest shutter speed) = 11.2
[/list]

Guess I could also do the above for SNR of 1 as the acceptable lower end.

But, 11.2 (lowest acceptable SNR = 2)... pretty close to 11.7 (lowest acceptable SNR = 1).

Will try w/ friend's D7000 this weekend... then D800 soon (hopefully).
[/quote]

11.2 is actually awfully close to what the other method gets for the 100% view (it seems like you are doing 100% view with the wedge)
 
Upvote 0
Re: DxO results out for 5D3

jrista said:
Right, sorry. I know LTRLI's formula is not a downsampling formula, what I was trying to say is I don't think it takes into account the realities of downsampling properly. To put it another way, I think it produces a result without context, which is why it seems to produce much higher numbers for DR than would seem realistic when computing DR for a downsampled image. In its own right, taken at face value (without any context) its certainly not an invalid formula...but do the results produced by it have real-world applicability? Do they represent something real, or is the formula too simplistic? I used to believe that DxO's Print DR numbers, which are derived the same way, were simply demonstrating that once you eliminate noise, you are then realizing the sensors full potential. Kind of tough to keep believing that, though, when the D800's Print DR numbers are BETTER than the sensors full potential...its not quite like a star trek warp drive...you don't get 120% of maximum out of a 14-bit sensor. ;)

Imagine you have 32MP sensor and you then averaged every 16 pixels into 1 and got 2MP output. And you then measure the Std Dev of the black frame. It's measure lower from the 2MP version than from the 32MP version but the white frame max saturation will measure the same in both cases. Plug into the DR formula and it goes up. It's not magic because the 2MP file also shows a heck of a lot less detail than the original 32MP file did so you traded that detail away for the higher score on the other aspect in a sense, what you are really trying to do it just compare the two files at the same scale. Even if the simplistic normalization didn't quite match in practice and over-stated something a little, the thing is in practice used use advanced NR to equalize them which would be doing better anyway, so there would be no problem there at all.


As for SNR stuff they want to compare at the same noise scale otherwise you might be comparing a 32MP file that has lots of high frequency noise to a 2MP file that all the higher powers of noise automatically clipped away so it's not fair to compare the two images as if they were at the same noise scale. So they are really just normalizing the power scale of the noise between the two to not unfairly make the higher MP cameras look worse. You are reducing noise and signal so it's not like the image really improves any, it's actually a worse output really, yeah but you are just trying to do relative noise comparisons here and to compare it to the other file fairly that already had all the high frequency noise clipped away otherwise you are comparing the total noise over a larger frequency range than for the other one where the upper end of the range was hard clipped off which isn't fair. To do it well (not talking in terms of processing images for use but just for this relative comparison) you want to first gaussian blur away the high freq noise and then downsample to the scale you blurred it away to. or maybe more simply upscale the lower MP cam to the match the higher. Or you could just look at both at 100% on a monitor but stand back farther from the screen for the higher MP camera whatever distance is needed to make it's detail the same as what the lower one can capture. i think i garbled that a bit


Do you think the D700 has much better ISO1600 than the 5D2? Because if you don't normalize, and you now seem to say that normalization doesn't make sense and isn't fair, then the D700 will stomp your 5D2, when the reality is it isn't fair to say that it does.
 
Upvote 0
Re: DxO results out for 5D3

jrista said:
Wow! So odd... ;) I definitely question the DR formula...although it is interesting that you got 13.2 EV. I should have actually run that calculation myself to prove earlier points...that there is some biased skew between Canon Print DR results and Nikon Print DR results. Why is it that Nikon images gain so much more than Canon sensors when doing Print DR?

Not all of them do. The D800 gains more because it has a ton of MP, 36.

OTOH compare the D700 to the 5D2 and then the Canon gains more, because it has a ton of MP 21.




Perhaps, but the major differences only seem to exist in normalized results. The two brands are a lot closer in performance when comparing native results...which makes me suspect that there may be some kind of software-level bias involved, rather than significant hardware-level differences. There is definitely a value to producing normalized results for consistency in comparison...but DxO is a bit of a black box. They expose some details and a little bit of math about their process, but there are also a lot of unknowns. Involving software in the process can affect results in a LOT of ways that could affect the objectivity of the final outcome. Who knows, for sure, exactly what their software actually does. Does it have any brand-specific optimizations that aren't publicized? If so, how are they skewing results?

it's just the MP scale

if you compare a 9MP cam print vs screen there it will look almost the same, if you compare a 32MP print ot screen it will look different since it is farther from their 8MP target.

if you compare a 32MP cam and a 6MP cam the relative difference print vs screen there will be a lot more than if you compare a 32MP to a 30MP cam or a 21MP to a 10MP, etc.

The ironic thing here is that you can completely ignore the normalized results, and the native hardware results still indicate that Nikon and Sony sensors are technologically superior. Thats all someone really needs to say, and its not something that can really be (nor should be) disputed...but the Print DR results and funky S___ like "bonus points" are involved in the final "scores" that DxO gives to each camera. I find that to be fishy as well, and unnecessary. You don't need biased results to demonstrate that Sony Exmor based cameras offer superior IQ, and the actual meaningful differences between the best of the vest (D800) and the best of the competition (5D III or any one of the measured Digital MF cameras) are not really as big as DxO is making them out to be. Thats where my pet peeve lies...bias and skew when its unnecessary and not really honest or helpful to potential customers of any brand.

Once again you need to compare image at the same scale other you compare noise at different frequencies as if they were the same. And don't forget that there are other cases where it then 'biases' in favor of Canon, say comparing 5D2 vs D700.
 
Upvote 0
Re: DxO results out for 5D3

And to not bubble burst too much so once again it is awesome they gave it 1 series AF and increased fps and all, it's quite a beast and handles way better than the 5D2 from what I read, and well much of that is simply fact, 6fps is faster than 4fps and the faster shutter response etc are fact, AF is a better on paper, didn't try it yet myself, reports sound very, very good so far though. So for non-liveview tripod shooting it should work anywhere from a little bit better to vastly better depending what you are doing with it.

And the SNR is tied for best ever which is all that we could have hoped for there (at expense of a bit of color-blindness which is why DxO scores it lower for high iso, although their weighting probably exaggerates the usable difference for SNR/DR alone, sorting out the color-blindess effects is really tricky, anyway DxO just weighs that in heavily as an incentive so that one makers doesn't just totally thin it out while others don't and make the ones who don't look bad, you can argue what the weighting should be).

The DR was already known to be coming in no better than the 5D2. So nothing new here.

The SNR part of results here are something positive though since there were some questions about exactly where it would end up. The early talk, in this case, was just best guesstimates. Some had said it might only be 1/4 to 1/3 stop better, i.e. nothing doing, so the results here showing it to be around 1/2 to 2/3+ better depending upon the ISO are pleasant.

It was nice to see it ended up at the high end of the early predictions and matching the D800/D4/D3s.

(and real world, since the D3s has much lower resolution I think you can say it would often look worse at high ISO than the others, if you ignore DR where it has a full stop better than D800 and 5D3, so it's darker shadow areas will look better at high iso and you won't feel quite so clipped in with teh dynmaic range, but the lower res means clumpier larger sized noise and less detail which tends to look uglier and doesn't take advanced NR as well so overall for scenes not having tons of deep shadows it probably wouldn't look quite as good as the others at high iso actually, although for certain scenes it might look better than the D800 and 5D3)

The people suggesting the lesser SNR improvement for the 5D3 over the 5D2 were failing to take into account that the 5D3 rates ISOs more conservatively than the 5D2 does I think.

Not that 1/2 to 3/4 stop is insanely OMG but it's realistically all we could have hoped for (nobody has done better yet in any typical consumer camera). And the high iso banding is also lesser in the 5D3 than the 5D2. You also get 1/2-2/3 stop more DR at the much higher ISOs than the 5D2.

Anyway so long as you are not a 100% tripod/liveview shooter the 5D3 certainly improves tons of things compared to the 5D2.








But it would be nice if Canon finally paid some attention to low ISO DR and image quality. They haven't done a single improvement there since the 1Ds3 (which is arguably still the best Canon, by a tiny trace, at ISO100).
(I saw a tiny bit of talk from some with bit of Canon connection say that they got the impression that Canon was apparently taken by surprise that Nikon/Sony decided to push their FF exmor stuff out the door so quickly and they didn't think they had to push their FF sensors yet and though they could focus on improving body handling mostly for now. As some have been saying, Canon management was acting way too complacent and king of the hill resting on their laurels. I also saw some disturbing talk that they seem to think it was only the MP that are exciting about the D800 and they still don't get the dynamic range stuff, not sure if that is true, but at least a few seem to have that unfortunate, IMO, point of view and that they, or at least a few (few? some? many? who knows) might consider a MP count higher than the D800 a huge success even if it had the same old 2007 dynamic range again. Anyway that is somewhat hearsay so who really knows exactly what they are thinking.)
 
Upvote 0
Re: DxO results out for 5D3

LetTheRightLensIn said:
Normalization. I do not think it means what you think it means. ;D

Since you seem to be the self-proclaimed go-to guy regarding normalization, you can probably easily answer this. Suppose you have a sensor that captures 1Gpix of data with DR of 2 stops. You expose the highlights to LV+15 and thus record luminosities between LV+13 and LV+15. How do you suppose "normalizing" this data will somehow generate luminosity values below LV+13 and/or above LV+15?
 
Upvote 0
Re: DxO results out for 5D3

straub said:
LetTheRightLensIn said:
Normalization. I do not think it means what you think it means. ;D

Since you seem to be the self-proclaimed go-to guy regarding normalization, you can probably easily answer this. Suppose you have a sensor that captures 1Gpix of data with DR of 2 stops. You expose the highlights to LV+15 and thus record luminosities between LV+13 and LV+15. How do you suppose "normalizing" this data will somehow generate luminosity values below LV+13 and/or above LV+15?

Why would the sensor suddenly not record any data lower than 2 stops below LV+!5? It would. It would be a mess of noise at 1GP scale though but it would still grab the data.
 
Upvote 0
Re: DxO results out for 5D3

LetTheRightLensIn said:
straub said:
LetTheRightLensIn said:
Normalization. I do not think it means what you think it means. ;D

Since you seem to be the self-proclaimed go-to guy regarding normalization, you can probably easily answer this. Suppose you have a sensor that captures 1Gpix of data with DR of 2 stops. You expose the highlights to LV+15 and thus record luminosities between LV+13 and LV+15. How do you suppose "normalizing" this data will somehow generate luminosity values below LV+13 and/or above LV+15?

Why would the sensor suddenly not record any data lower than 2 stops below LV+!5? It would. It would be a mess of noise at 1GP scale though but it would still grab the data.

Ah, I should have also mentioned that the camera would have 2-bit ADC. Which then means that in this case anything below LV+13 is recorded and interpreted as LV+13. It is a silly hypothetical camera, but the principle is the same for any N-bit ADC--you simply don't have any data beyond the range of the ADC. And no amount of normalization is going to change that.
 
Upvote 0
Re: DxO results out for 5D3

straub said:
LetTheRightLensIn said:
straub said:
LetTheRightLensIn said:
Normalization. I do not think it means what you think it means. ;D

Since you seem to be the self-proclaimed go-to guy regarding normalization, you can probably easily answer this. Suppose you have a sensor that captures 1Gpix of data with DR of 2 stops. You expose the highlights to LV+15 and thus record luminosities between LV+13 and LV+15. How do you suppose "normalizing" this data will somehow generate luminosity values below LV+13 and/or above LV+15?

Why would the sensor suddenly not record any data lower than 2 stops below LV+!5? It would. It would be a mess of noise at 1GP scale though but it would still grab the data.

Ah, I should have also mentioned that the camera would have 2-bit ADC. Which then means that in this case anything below LV+13 is recorded and interpreted as LV+13. It is a silly hypothetical camera, but the principle is the same for any N-bit ADC--you simply don't have any data beyond the range of the ADC. And no amount of normalization is going to change that.

But the way you are reading it out doesn't make sense since it wouldn't store LV 13-15 from scene.
 
Upvote 0
Status
Not open for further replies.