DxOMark Sensor Performance: Nikon vs. Canon

[mystic_theory]

It's amazing how much distance in DxO scores there is between Nikon sensors and the corresponding Canon ones: I agree that the overall score gives just a rough estimate of the overall quality of the sensor, but what worries me are the differences in high ISO noise and DR, which are instead very precise and reliable measurements.

 

[sarangiman]

...
That said, DxO is a complete and total joke. This is a company that ranks consumer DSLRs above medium format digital backs in IQ.
...

There is no rule that says a MF digital back must be better at taking pictures than a DSLR.

If the design and technology used by the DSLR is superior to that of the digital back and delivers better images then it stands to reason that the sensor can be rated above a MF back.

+1

Thank you dilbert for stating what with common sense & logic should be apparent to everyone.

 

[sarangiman]

I have said this many times. You CAN NOT compare dynamic range on ISO100, and assume everybody works in ISO100, and that this is the correct ISO to measure by, like DxO does. I RARELY go below ISO 1600 for my kind of work.

You do realize that at base ISO, the D800 has more than 2.5EV more DR than the 5DIII, and at worse, i.e. at ISO 25,600, it has 0.26EV worse DR than the 5DIII. So, compare at whatever ISO you'd like. The D800 is rarely going to do worse than the 5DIII, but can do much better.

Furthermore, the future is likely 'ISO-less'. In other words, ISO becomes metadata written into the file, much like white balance. Amplification is done during RAW processing. That way, you actually retain the DR of base ISO even at high ISO (w/ low enough read noises).

We just need manufacturers (both hardware & software) to implement this. Personally, I would've already implemented this w/ the D800, given the low read noise of ~3 e-.

But maybe the industry is waiting for a read noise of 1 e-, or a 16-bit ADC.

 

[Fishnose]

...
That said, DxO is a complete and total joke. This is a company that ranks consumer DSLRs above medium format digital backs in IQ.
...

There is no rule that says a MF digital back must be better at taking pictures than a DSLR.

If the design and technology used by the DSLR is superior to that of the digital back and delivers better images then it stands to reason that the sensor can be rated above a MF back.

+2

 

[jthomson]

BTW! Anyone find it strange DxO has the finished analysis/review of the D600 days after it is released? And they still haven't finished they analysis/review of the 1D-X !?
...

Nope.

Nikon had the D600 on sale via the web/stores 5 days after it was announced, so it is likely that Nikon were able to ship a production matched D600 to DxO ahead of time.

How long did it take Canon to even get 1DX cameras out to folks that were going to need them for the Olympics? And how long did it take for the 1DX to show up in stores? Don't blame DxO for Canon's inability to delivery/perform.
[/quote]

DXO appears to have a Nikon bias. They haven't released scores for the 1DX, the T4i or the Olympus OMD EM-5, but they get the D600 out within days of its release.

 

[Fishnose]

The rub here is that if you're Canon and you already know that your camera's sensor isn't going to score better than anything Nikon has published then you're not going to be in any sort of rush to have it tested by DxO. I don't know if DxO purchases the cameras for themselves or waits for vendors to send them freebies...

DxO being a rather small company, I would guess they are in the position of getting access to test subjects in a few ways:
1. Manufacturer or distributor sends them a copy to test (much like magazine/web reviewers are sent copies)
2. The company buys one, as the item in question is central to their lens testing
3. Someone on the staff buys one
4. They borrow one from someone else

If none of the above fits, they don't test it.

 

[HarryWintergreen]

Sensor performance isn't everything...

Not only not everything, but far less than even remotely close to everything. Consider all the glass the light must pass through before it even reaches the sensor.

Also consider that DxOMark is evaluating only the sensor, and also that their "Overall Score" is composite of three arbitrarily chosen "Use Case Scores" that are combined in a 'weighted' manner, but the weighting is not disclosed. Furthermore, their use case scores are normalized to an 8 MP file size, which explains how a camera with 14 bits per pixel can, according to DxOMark, actually deliver a dynamic range greater than 14 bits of EV.

IMO, their Measurements (screen) are valid and quite useful. Their Scores are steaming pile of misleading cow excrement.

Just a little reality check...

+1 (a bit late, I know). I wished any limitations referring to my photographic skills could be ascribed to dynamic range and what have you. ;D

 

[Loswr]

...
That said, DxO is a complete and total joke. This is a company that ranks consumer DSLRs above medium format digital backs in IQ.
...

There is no rule that says a MF digital back must be better at taking pictures than a DSLR.

When did DxO start evaluating the ability of a camera to take pictures? That statement implies lots of performance characteristics that DxO doesn't measure. Their measurements are for specific aspects of sensor performance, as they define them (obviously, I'm referring to their sensor measurements, not their lens measurements).

DxO is also capable of telling you how lenses perform on various cameras, e.g. the 70-300L:
http://www.dxomark.com/index.php/Lenses/Camera-Lens-Database/Canon/EF70-300mm-f-4-5.6L-IS-USM/(camera)/436

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.

 

[Viggo]

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.

Noooo? Are you kidding? Really, lower for the mk2? LOOOOL . That adds to the credibillity.... :

 

[itsnotmeyouknow]

http://gearburn.com/2012/08/canon-eos-1dx-review/

DxOMark doesn't mean anything...

Canon still better, specially in HI ISO...

I have the D800 and the 5D3, and I would agree that the Canon does better at high ISO. For a predominantly landscpe shooter like myself, though the amount of banding at lower ISO in combination with not great DR renders the camera a handicap to me. I have some great lenses that are very sharp but they have one hand tied behind their back by the banding issue. I can push the shadows on a D800 shot and keep noise completely at bay and you have to push it way too far to get the chroma noise you get from the 5D3....I'm not a pixel peeper and I hate arbitrary resolution charts. I judge on the image as a whole. In some instances the banding on 5D3 is visible at 1024 px. Not Good Enough for a camera that cost me £3200. I can't live with that and whereas I had no alternative before, I do now. This is what will cost Canon money. I have 10 L lenses. Yet I still feel the need to move to the Nikon d800. Whether it is a permanent move depends on what Canon does next. I am going to be selling the glass where I have overlaps and 3 of these will be L lenses. I'll use the proceeds to get good zooms on Nikon.

 

[Zlatko]

I have the D800 and the 5D3, and I would agree that the Canon does better at high ISO. For a predominantly landscpe shooter like myself, though the amount of banding at lower ISO in combination with not great DR renders the camera a handicap to me.

I do weddings and portraits with the 5D3 and have never seen any banding issue.

 

[sarangiman]

I have said this many times. You CAN NOT compare dynamic range on ISO100, and assume everybody works in ISO100, and that this is the correct ISO to measure by, like DxO does. I RARELY go below ISO 1600 for my kind of work.

Just because this is brought up rather often, allow me to point out:

• At ISO 100: D800 has >2.5EV greater DR than 5DIII.
• 
  At ISO 12,800, D800 has 0.09EV worse DR than 5DIII, probably below the margin of error in measurement.
  
• 
  At ISO 25,600, D800 has 0.26EV worse DR than 5DIII, probably still within the margin of error in measurement.
  

Full measurements here: http://bit.ly/OEUgZY

So now matter how you look at it, D800 is rarely going to perform worse in DR than the 5DIII, if at all, but can perform much much better than the 5DIII.

When camera/software goes ISO-less, we'll see cameras maintaining base ISO DR even at high ISOs. Imagine >13EV DR at ISO 6400!

 

[sarangiman]

I do weddings and portraits with the 5D3 and have never seen any banding issue.

I see banding in my 5DIII images just from having Lightroom automatically correct the vignetting for my 24/1.4 & 35/1.4 lenses. I just try to ignore it. :'(

 

[Loswr]

I do weddings and portraits with the 5D3 and have never seen any banding issue.

I see banding in my 5DIII images just from having Lightroom automatically correct the vignetting for my 24/1.4 & 35/1.4 lenses. I just try to ignore it. :'(

Have you tried DPP instead of LR?

 

[mystic_theory]

I have said this many times. You CAN NOT compare dynamic range on ISO100, and assume everybody works in ISO100, and that this is the correct ISO to measure by, like DxO does. I RARELY go below ISO 1600 for my kind of work.

Just because this is brought up rather often, allow me to point out:

• At ISO 100: D800 has >2.5EV greater DR than 5DIII.
• 
  At ISO 12,800, D800 has 0.09EV worse DR than 5DIII, probably below the margin of error in measurement.
  
• 
  At ISO 25,600, D800 has 0.26EV worse DR than 5DIII, probably still within the margin of error in measurement.
  

Full measurements here: http://bit.ly/OEUgZY

So now matter how you look at it, D800 is rarely going to perform worse in DR than the 5DIII, if at all, but can perform much much better than the 5DIII.

+1

I get tired of reading a pile of fuming BS to cover up the 5DIII shortcomings compared to the D800.

 

[DB]

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.

DxO have actually responded to this anomaly on their website when questioned by Canon 70-200mm f2.8 IS II users as to why the apparently superior mark II scores so much lower than its predecessor and their reply was that the older lens has better resolving power (@ 62 line pairs per millimeter vs 52 lp/mm for the new lens). But their website also states that there is enormous variation in resolution throughout the focal range, plus their Resolution Index scores are based on averages of multiple shots taken typically between f5 and f8 throughout the zoom range.

Clearly then, one can challenge the validity & reliability of this 'Resolution' test by conducting a similar test (they outline in graphic detail their testing procedure) using say multiple samples (half a dozen) of each of the 2 lenses in question (to also test if copy-to-copy variation exists).

Secondly, irrespective of the first answer, what the results suggest given that the Mark II surpasses the Mark I in each of the other sub-categories (especially CA), is that their arbitrary weighting (Black Box is an apt description as they do not divulge their weighting methodology) system is heavily skewed towards 'Resolution' when computing the Overall Score.

The very fact that they place more emphasis on one single category (more so than all other sub-categories combined) is evidence that their overall score is biased & subjective. According to DxO the most impressive Canon lens in the entire EF range is the 85mm f/1.8.

What they should do is post the summary stats of all the resolution test results in a table (freq dist along with standard deviation figures) - that way we can see if it is case of some outlier or skew (to wide or tele) that is causing them to conclude that the mark I is better.

No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

 

[Loswr]

I get tired of reading a pile of fuming BS to cover up the 5DIII shortcomings compared to the D800.

Honestly, I think the bottom line is that both produce excellent images - if testing to the nth degree is needed to see differences, you have to question the significance of those differences.

Are there shortcomings? Yes. IMO, the biggest shortcoming of the 5DIII relative to the D800 is that the former doesn't work with the excellent 14-24/2.8 and an actually available 200-400/4. Likewise, the biggest shortcoming of the D800 relative to the 5DIII is that it doesn't work the MP-E 65mm, TS-E 17mm, and Canon's slightly better superteles. Of course, if you're willing to give up AF, Nikon lenses can be mounted on Canon bodies, whereas the converse isn't true...

No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

+1, and that's my beef with DxOMark.

 

[bdunbar79]

Show up to a fast-action soccer game with a D800 and then see which one, the 5D Mark III, or the D800, really has the shortcomings. Kick off is at 7pm under the lights.

 

[Zlatko]

According to DxO the most impressive Canon lens in the entire EF range is the 85mm f/1.8.

... which is freaking bizarre. The 85 f/1.8 is very fine lens, but there is NO WAY that it is the most impressive lens in the entire EF range. That tells me a lot about DxO. When everyday experience informs you better than the DxO score, then the DxO score lacks credibility.

 

[Fishnose]

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.

DxO have actually responded to this anomaly on their website when questioned by Canon 70-200mm f2.8 IS II users as to why the apparently superior mark II scores so much lower than its predecessor and their reply was that the older lens has better resolving power (@ 62 line pairs per millimeter vs 52 lp/mm for the new lens). But their website also states that there is enormous variation in resolution throughout the focal range, plus their Resolution Index scores are based on averages of multiple shots taken typically between f5 and f8 throughout the zoom range.

Clearly then, one can challenge the validity & reliability of this 'Resolution' test by conducting a similar test (they outline in graphic detail their testing procedure) using say multiple samples (half a dozen) of each of the 2 lenses in question (to also test if copy-to-copy variation exists).

Secondly, irrespective of the first answer, what the results suggest given that the Mark II surpasses the Mark I in each of the other sub-categories (especially CA), is that their arbitrary weighting (Black Box is an apt description as they do not divulge their weighting methodology) system is heavily skewed towards 'Resolution' when computing the Overall Score.

The very fact that they place more emphasis on one single category (more so than all other sub-categories combined) is evidence that their overall score is biased & subjective. According to DxO the most impressive Canon lens in the entire EF range is the 85mm f/1.8.

What they should do is post the summary stats of all the resolution test results in a table (freq dist along with standard deviation figures) - that way we can see if it is case of some outlier or skew (to wide or tele) that is causing them to conclude that the mark I is better.

No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

One very good reason that resolution should be weighted quite heavily is that resolution is pretty much impossible to 'fix in the mix'. Okay, a bit of sharpening, but that's not optimal as a solution for lower resolution.

Other issues such as CA, vignetting and distortion, on the other hand, can be fixed very well. And that is precisely the business DxO are in.