DxOMark Sensor Performance: Nikon vs. Canon

[Meh]

Everybody plays with the same laws of physics: I'd be very surprised if Canon does not have high-DR tech in their back pocket.

Those pesky laws of physics always getting in the way. Not sure if Canon would have some advancement that they're holding back but it could be the sensor engineers have not been given the mandate/budget to improve low-ISO DR or they find it sacrifices something else. Hard to say.

 

[Meh]

I'm just surprised at how many people call foul at DXO without taking the effort to really analyze what the measurements/results show.

+1

confirmation bias, thats why you didnt also get a response over your comment @ dtaylor and his "the right way" DR test.

+1 confirmation bias is detestable!

 

[Meh]

Sensor performance isn't everything...

Not only not everything, but far less than even remotely close to everything. Consider all the glass the light must pass through before it even reaches the sensor.

Also consider that DxOMark is evaluating only the sensor, and also that their "Overall Score" is composite of three arbitrarily chosen "Use Case Scores" that are combined in a 'weighted' manner, but the weighting is not disclosed. Furthermore, their use case scores are normalized to an 8 MP file size, which explains how a camera with 14 bits per pixel can, according to DxOMark, actually deliver a dynamic range greater than 14 bits of EV.

IMO, their Measurements (screen) are valid and quite useful. Their Scores are steaming pile of misleading cow excrement.

Just a little reality check...

That's about right ;D

 

[sarangiman]

+1

confirmation bias, thats why you didnt also get a response over your comment @ dtaylor and his "the right way" DR test.

Well, let's give @dtaylor some more time to respond. If I remember correctly, he said he uses transmission wedges for his tests, so I'd be very surprised if he got different comparative results between bodies. Absolute results may vary based on what you set as your acceptable SNR on the low end. But even then, the D800 should consistently show ~2.5EV higher DR (pixel-level) than 5DIII. I've tested this numerous times, b/c I was also incredulous at first.

I even remember being incredulous about the D800 beating the D4 in DR, but the data is the data. The D800, despite having a lower saturation capacity per-pixel, simply has lower read noise on a per-pixel basis. Low enough that its DR even trumps the D4. That's also why it's not hard to imagine the D800 beating medium format in DR (though I agree with neuroanatomist that I'm not sure how fair the normalization formula is | e.g. D800 @36MP goes up 1.1EV from screen to print, Hasselblad @50MP goes up 1.35EV, D600 @24MP goes up 0.8EV... I really can't comment one way or the other if it's fair as I haven't delved into their normalization formula yet).

Interesting point about DXO getting the D600 scores out before the 1Dx. Then again, if I had a D600 vs. a 1Dx in my hands, I'd be more excited about testing the D600 b/c the 1Dx has already been shown to not have much more DR than the 5DIII (e.g. here: http://home.comcast.net/~NikonD70/Charts/PDR.htm#EOS%205D%20Mark%20III,EOS%201D%20X)

 

[jthomson]

Strange that they can get the Nikon D600 out on the day that it is released but haven't yet got around to the 1DX.

That speaks to an entirely different potential bias, but without knowledge about the rationale for the discrepancy, no conclusions can be drawn. Maybe Nikon overnight shipped them the first D600 off the line, and they're still waiting on a retail back ordered 1D X. Heck, noted bird photographer and Canon Explorer of Light, Art Morris, just recently got his 1D X - I've had mine longer than him.

Good point about the availability of the 1DX but I was just at their site and they don't even have a rating for the T4i/ 650D.

 

[Zlatko]

A lot of denial in the forum about inferiority of Canon's sensors. The knee jerk argument is that Dxomark:
1) is biased, or
2) their tests are bad because "I" get superior results out of my Canon, unlike "my" Nikon friend who is struggling, or
3) is meaningless because I can bracket and get high DR, so low DR is not big deal

DXOMarks is simply giving empirical evidence to the inferiority of Canon's decade old tech versus modern tech of Sonikon. If we as Canon consumers (whether we like the brand or, more importantly, stuck to it due to sunk costs) live in denial, Canon has no incentive to improve. Result? we get shafted while competition has better cameras. Period!

The problem with DxO's scoring is that it doesn't match what I can see with my own eyes. Last year I shot two Canon 5D Mark II's and two Nikon D7000's. The Canon is in a higher price category, but DxO actually scores the D7000 higher:

Canon 5D Mark II score = 79
Nikon D7000 score = 80

So the D7000 sensor is better, right? Not at all. Just compare the color in portraits from each camera and judge which one shows humans looking more like actual humans. The 5D Mark II's won easily; the D7000's were actually worse in color and more prone to blowing highlights. Of course, that result is to be expected based on the rather big price difference, but DxO would have you believe the those cameras' sensors are a close match and that the D7000's is a tiny bit better. Well, they are very wrong.

So whatever DxO is measuring, they aren't measuring everything that's important to me or scoring in a way that matches my experience. I heard a similar experience from a wedding photographer who had moved to a higher end Nikon and missed the color of his 5D Mark II.

 

[Meh]

1. their overall scores are weird, how do you combine so many different performance factors into one number that would work for everyone? you can't

Precisely. They have arbitrarily designed one of many possible algorithms to combine their various measurements into a "score".

 

[sarangiman]

The 5D Mark II's won easily; the D7000's were actually worse in color and more prone to blowing highlights.

A camera being prone to blowing highlights says more about its metering than its dynamic range. The dynamic range of the D7000 is demonstrably worlds above that of any of the 5D cameras. Yes, despite its price. It's all about saturation well capacity & read noise (& megapixel count of course, if we're talking about normalized results).

 

[Zlatko]

agree...if you look at the scores of dxo on low light iso the 1d mark 3 is destroied by a d5100; but maybe i am blind

There is something absurd about those scores. If the D5100 is anything like the D7000, then I would much prefer the sensor of the 1D Mark III.

 

[Meh]

Question for you: my understanding of Bayer pattern sensors is that a group of 4 monochrome {R|G|B} are de-mosaiced to form one color RGB pixel. If I combine two 14-bit wells I get (roughly) one 15-bit well. If I combine four 14-bit wells I should get (roughly) one 16-bit well. Certainly they wouldn't do a crude addition of wells, but it seems very plausible that you can get >14EV DR, even without re-normalizing. What's wrong with this interpretation?

I don't believe it's correct to say that 4 are combined into 1. The de-mosaicing is an interpolation algorithm that uses the values from adjacent pixels to estimate the intensity and color of each pixel. The demosaicing algorithm doesn't do any binning in the way you suggest. IIRC the Canon C300 does perform the binning that way to give better color accuracy for each pixel.

 

[Zlatko]

The 5D Mark II's won easily; the D7000's were actually worse in color and more prone to blowing highlights.

A camera being prone to blowing highlights says more about its metering than its dynamic range. The dynamic range of the D7000 is demonstrably worlds above that of any of the 5D cameras. Yes, despite its price. It's all about saturation well capacity & read noise (& megapixel count of course, if we're talking about normalized results).

As I wrote, the problem was the color in portraits was distinctly worse. The blown highlights were an added problem. I'm talking about using the cameras for actual photography, not lab tests. So, yes, consistent with its price, the D7000 was distinctly worse in actual photography. The D7000 was a bit like a kid who scores great on a standardized test, but is lacking in essential people skills.

 

[gilmorephoto]

The 5D Mark II's won easily; the D7000's were actually worse in color and more prone to blowing highlights.

A camera being prone to blowing highlights says more about its metering than its dynamic range. The dynamic range of the D7000 is demonstrably worlds above that of any of the 5D cameras. Yes, despite its price. It's all about saturation well capacity & read noise (& megapixel count of course, if we're talking about normalized results).

As I wrote, the problem was the color in portraits was distinctly worse. The blown highlights were an added problem. I'm talking about using the cameras for actual photography, not lab tests. So, yes, consistent with its price, the D7000 was distinctly worse in actual photography. The D7000 was a bit like a kid who scores great on a standardized test, but is lacking in essential people skills.

+1

 

[sarangiman]

As I wrote, the problem was the color in portraits was distinctly worse. The blown highlights were an added problem. I'm talking about using the cameras for actual photography, not lab tests. So, yes, consistent with its price, the D7000 was distinctly worse in actual photography. The D7000 was a bit like a kid who scores great on a standardized test, but is lacking in essential people skills.

I understand, but my point is simply that the 'blown highlights', in this case, is not indicative of anything wrong with DXOs sensor scores for the D7000. The (Sony) sensor in the D7000/D5100 really does have significantly more DR than any Canon sensor.

If you found that it blows highlights too often, then perhaps you disagree with Nikon's metering algorithms in those situations.

As for Nikon/Canon color, I'm inclined to agree with you & say my personal taste is for Canon's default colors. But I bet if you really wanted to you could build a DNG profile that makes a Nikon camera match Canon's default colors. Whether or not you wish to do so is another matter altogether.

 

[Martin]

Actually I think that canon makes everthing to postpone the 1dx scoring by dxo. It would be funny if d600 scored better than flagship canon. Unfortunately thats probably the reality.

 

[weixing]

Hi,
IMHO, DxOMark is not testing the sensor, but the camera itself because they use the RAW file produce by the DSLR (they didn't remove the sensor from the body and test, right?) and if the RAW file produce by the camera is not the RAW data, then it's not testing the sensor and the sensor compare between brands is somehow not valid. I believe all RAW file is not RAW data and Nikon RAW file are well know for it.

Anyway, I was wondering how do Nikon D800 and D600 get a dynamic range of more than 14EV when there is only 14-bit of data? Compress the dynamic range or non-linear data?

Have a nice day.

 

[Orangutan]

I don't believe it's correct to say that 4 are combined into 1. The de-mosaicing is an interpolation algorithm that uses the values from adjacent pixels to estimate the intensity and color of each pixel. The demosaicing algorithm doesn't do any binning in the way you suggest.

I didn't mean to suggest binning at all; demosaicing is (presumably) a complex algorithm. I was making a simple point: if you take data from 4 photosites, each having 14EV of DR, you can demosaic in a way that gets you more than 14EV DR in the resultant pixel. It's not guaranteed that you would, but it's possible.

 

[Meh]

I don't believe it's correct to say that 4 are combined into 1. The de-mosaicing is an interpolation algorithm that uses the values from adjacent pixels to estimate the intensity and color of each pixel. The demosaicing algorithm doesn't do any binning in the way you suggest.

I didn't mean to suggest binning at all; demosaicing is (presumably) a complex algorithm. I was making a simple point: if you take data from 4 photosites, each having 14EV of DR, you can demosaic in a way that gets you more than 14EV DR in the resultant pixel. It's not guaranteed that you would, but it's possible.

I was addressing only the first line of your statement where you said about demosaicing that 4 pixels are made into one RGB pixel which is not correct the way you wrote but fair enough if that's not what you meant.

Binning pixels still doesn't result in greater bit-depth though because that is limited by the ADC. Each pixel can only be assigned 14-bit luminosity regardless of how many electrons are in the well. For example some FF sensors have photosites with say 50,000 electron full-well capacity. 14-bit addressing only requires about 16,000 electrons so you have about 3 electrons per level. Binning 4 pixels gives you more pixels but you still only have 14-bit addressing in the ADC.

But I'll leave it there since it's been a long time since I've been involved with that kind of thing and may already not be explaining it quite right.

 

[LetTheRightLensIn]

Hi,
IMHO, DxOMark is not testing the sensor, but the camera itself because they use the RAW file produce by the DSLR (they didn't remove the sensor from the body and test, right?) and if the RAW file produce by the camera is not the RAW data, then it's not testing the sensor and the sensor compare between brands is somehow not valid. I believe all RAW file is not RAW data and Nikon RAW file are well know for it.

Anyway, I was wondering how do Nikon D800 and D600 get a dynamic range of more than 14EV when there is only 14-bit of data? Compress the dynamic range or non-linear data?

Have a nice day.

1. Why would they test the sensor removed from the camera? Can you get a picture out of a bare sensor? It's the RAW file that the user gets to use. Most of DSLR RAWs are pretty RAW under standard conditions and settings (Nikon does cook books starting with longer, but not long enough IMO, exposures, etc.).

2. They get more than 14EV on the normalized comparison, it's normalized to 8MP so with 36MP to play with there is a lot of room to get the extra bits after normalization there.

 

[sarangiman]

2. They get more than 14EV on the normalized comparison, it's normalized to 8MP so with 36MP to play with there is a lot of room to get the extra bits after normalization there.

Exactly. Say you have 14EV DR at the pixel level; i.e. you've perfectly matched your ADC to the DR attainable at the pixel level (e.g. saturation capacity = 16,384 e- & read noise = 1 e-). But say this is for a 100MP sensor.

When you go to make a 4" x 6" print of that 100MP file, you'll be doing a lot of downsizing for a 300dpi printer. That process of downsizing will reduce noise. Therefore you will be able to lift shadows more while still producing an acceptable print. Therefore shadows with a pretty low SNR may suddenly become 'acceptable' the viewer of the 4x6 print.

Yada yada... that's how you get more dynamic range via normalization.

Although I still have a hard time wrapping my head around it.

Because if your SNR is ever <1, you're not going to magically recover the signal by resampling... resampling would only bring you closer to the average of the noise as SNR approaches 0.

But for more reasonable definitions of DR (e.g. Bill Claff's 'PDR' that defines the lowest acceptable SNR as 20), normalization expanding DR makes perfect sense.

 

[messus]

Canon has some catching up to do with respect to sensor performance as measured by http://www.DxOMark.com. Canon doesn't even come close to the top performing Nikons. (High score is better.):

Pts Model
=======
96 Nikon D800E
95 Nikon D800
94 Nikon D600
81 Canon 5D III
79 Canon 5D II

(The Canon 1Dx is not yet rated.)
What are the chances that one of the reasons for the new sensor in the 6D is to catapult Canon's sensor performance into the mid 90's? I can't see Canon doing that considering the $3,500 EOS 5D III just came out and has a score of just 81. But Nikon's new $2,100 D600 kicks butt with a score of 94!

Sensor performance isn't everything... but, if I were to choose Nikon or Canon today, I wouldn't be choosing Canon.

I have said this many times. You CAN NOT compare dynamic range on ISO100, and assume everybody works in ISO100, and that this is the correct ISO to measure by, like DxO does. I RARELY go below ISO 1600 for my kind of work.

BTW! Anyone find it strange DxO has the finished analysis/review of the D600 days after it is released? And they still haven't finished they analysis/review of the 1D-X !?

Well I don't find it strange at all, DxO is tight with Nikon, everybody should know that by now. I will not be surprised if they rate the 1D-X below the Nikon D3100, and neither should you! Do not expose your low intelligence by actually believing DxO is the standard for measuring camera low light and dynamic range performance.

The 5D3 has a MUCH better dynamic range, not to mention the low light ISO performance, than the D800 above ISO600.

Who cares what the Nikonioans down at DxO says, not me!

Have a nice day everyone!