DxOMark vs. Reality

[tnargs]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page, which is easily accesed from the About tab. Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance." Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure. But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

well spotted! Half way down the same page says "So before comparing cameras with Sensor Overall Score, it is important to first determine the resolution you are looking for (which largely depends on the size of the screen or the print you intend to use or produce). Once you choose an appropriate resolution, the Sensor Overall Score becomes a fair and powerful tool with which to make comparisons." My memory was pretty good from a couple of years back!

@tnargs, if anything, that just makes the main point of post #1 even stronger.

yes, my point

 

[tnargs]

....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

 

[Loswr]

What has Canon sales to do with DXO ?

Nothing. Which is exactly my conclusion:

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available. The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

If Consumer Reports gives one product a much higher rating that another product, that usually has a tangible impact on sales, i.e. those reports impact buying decisions. DxOMark has given Nikon higher ratings than Canon for years, and there does not appear to have been any impact of that on sales, i.e. they have no impact on buying decisions.

Given that:
a) there are flaws and ambiguity in their scores, not to mention some apparently aberrant results (e.g. 70-200 II),
b) their scores apparently (according to them) should not be used to compare cameras of different resolutions,
c) they are scoring only sensor performance, which is just one part of camera performance,
and d) the reduction of a complex imaging system to a single number is essentially meaningless anyway,

I would argue that not only do DxO's Scores have no impact on buying decisions in aggregte, they should have only a minor impact, if any at all, on personal buying decisions.

As I've pointed out before, their measurements are usually quite good (I say usually because of the above-referenced 70-200 II issue, where DxOMark are the only ones in the world who seem to think the MkI is better than the MkII, and chose not to test another copy but rather to defend their results, which really doesn't help their credibility). But their Scores are neither meaningful nor useful.

 

[PackLight]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.

Interesting you said 200mm, before I made that post it was the first lens I did look at. At f/2 the Nikon version is very bad in the corners. Supertele's are designed to be used wide open, wide open the Canon wins.

Since Canon doesn't have a 200-400mm yet, Nikon wins by default.

I compared the top Nikon and Canon 70-200mm before I made the last post, because I knew it is where you would want to go next. Again, in the side by side the Canon wins noticeably.

The 24mm f/1.4 versions of Canon and Nikon, Nikon is near a match but in no way wins. Even wider the 14mm f/.8 the Nikon lens is much softer. I would give Nikon the edge for wide angle, not because of its lenses but its sensor.

If you are comparing top of the line lenses to top of the line lenses Nikon doesn't win. In some categories it gets close to matching but not wining.

 

[straub]

I took this 5 min ago to show the difference between one of my 5dmk2 and d800

Congratulations, you are the first one to notice that D800 has more DR than 5D2 at base ISO. First prize is a set of tickets for Titanic on it's maiden voyage across the Atlantic.

 

[Maui5150]

I took this 5 min ago to show the difference between one of my 5dmk2 and d800

Ummmm Why did you not compare the d800 to a 5DMKIII at least.

Maybe I should test my 5DMKII versus my dad's D200 as a similar test. Any bets?

 

[aj1575]

I like that all of you make a little test. Go to dpreview and choose the review of a Canon Camera; a EOS650D for example (which has a very low DxO-Rating).
Towards the end of the review there is the comparison tool, you can now compare the EOS650D to every other camera tested on dpreview. Take for example a Sony SLT77, a Nikon D7000 and the Oly OM-D E M5. Now just play around with the tool a little bit, and deceide which one of the camera produces the best pictures. Then go to DxOMark and compare their scores there.
I found that interesting.

To my eyes the best sensor was the Nikon, but just a little bit ahead of the Oly, then comes the Canon with a little distance (even though I liked some aspects of the IQ better than from the Nikon, more details for example), and last is the Sony.

What is your impression? And even more interesting, how do they compare to the DxO-rating?

 

[Ellen Schmidtee]

Part of this is because lenses play a very important role in photography, and Nikon has lagged behind with FF lenses. They have very few really good lenses, their 14-24 being the best example followed by their new 85mm f/1.4. Their 24-70mm has horible CA, their 70-200mm is very good. There are a ton of "D" lenses that are good, but no one seems to want them, and they do not have coatings that are well suited to digital.

I'll bet that ignorance of DxO's (or any other specific site that does that kind of testing) existence plays an even larger part.

My two cousins bought Rebels, and I'll bet dollars to monopoly money that attempting this kind of discussion with them will make them give you the 'I'd rather not know what that is, and will you weirdow please go away?' look.

 

[jthomson]

....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

Confuse photographers and themselves. ;D

 

[unfocused]

....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

Confuse photographers and themselves. ;D

Best. Answer. Ever

 

[PackLight]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.

Interesting you said 200mm, before I made that post it was the first lens I did look at. At f/2 the Nikon version is very bad in the corners. Supertele's are designed to be used wide open, wide open the Canon wins.

Since Canon doesn't have a 200-400mm yet, Nikon wins by default.

I compared the top Nikon and Canon 70-200mm before I made the last post, because I knew it is where you would want to go next. Again, in the side by side the Canon wins noticeably.

The 24mm f/1.4 versions of Canon and Nikon, Nikon is near a match but in no way wins. Even wider the 14mm f/.8 the Nikon lens is much softer. I would give Nikon the edge for wide angle, not because of its lenses but its sensor.

If you are comparing top of the line lenses to top of the line lenses Nikon doesn't win. In some categories it gets close to matching but not wining.

I have tested lenses since 1978 and I have access to all Nikon and Canon lenses
Regarding 70-200/2,8, IS VR . Canons 70-200/2,8ismk2 lens is better than Nikons latest and it is seen mostly out in the corners .
200/2 they are close as they can be
200-400/4 Nikon excellent in a range of 50-100m and optimized for that range compare to infinity
400/2,8 Nikon sharper than Canons old 400, I have not tested the new one
600/4 excellent
Nikon 14-24 excellent but sensitive to flare, better at 14mm than Canons 14mm/2,8MK2 regarding resolution
35/1,4 Nikon overall better,slightly sombrero effect and better towards the edges and corners.
85/1,2 , Nikon 85/1,4 equal good, 85/1,2 lot of loCA, 85/1,8 canon excellent for the price , also nikon 85/1,8
Nikons 24-70 better than Canons 24-70 . Canon is is difficult to evaluate since it is a big difference amoung the copies, MK2 not tested

I think the new versions would be the standard to test against, since it is the current offering. That would apply to the 24-70 and the 400mm f/2.8.

I can see what you are saying with the 35mm, and I said Nikon has lenses as good on the wide end.

The 14-24 reported to be a very good Nikon lens, better than the 14mm though? A close match yes.

 

[MarioMachado]

the DxOMark is just nonsense, no way in this world a Nikon D300S is better than the 7D, even 40D.
Unfortunately where I work (cruise ship), they provide Nikon D300s and the camera sucks big time, poor in every thing. I hate touching this camera!

Canon, the only way!

 

[marekjoz]

Guys, don't you really have enough?

 

[Loswr]

now this was more a answer to the Neuro who finds it difficult to understand that DXO measurements also can be seen / reflected in an image.

Sorry, but what the hell gives you the idea that I don't understand that what DxO measures can be made to show up in images. Please try to comprehend the following:

• There are differences between Canon and Nikon sensors
• Some aspects of Nikon sensor performance are superior to Canon sensor performance
• Some aspects of Canon sensor performance are superior to Nikon sensor performance
• DxOMark attempts to quantify those differences and defines their own arbitraty priorities to reduce those measurements to 'Scores'
• DxOMark Scores are relevant only in specific situations, and those situations may or may not have any relevance for the user of a camera
• DxOMark Scores attempt to represent a complex system with a single numeric comparison, an effort which is basically doomed to failure from the start, and is ultimately meaningless

So, if you routinely find yourself in situations where you must push shadow exposures 4 stops with less noise impact, you should probably be shooting Nikon (or asking how you manage to get yourself into those situations in the first place..). If you want to handhold your 600mm lens, you should probably be shooting Canon (or use a 300mm lens with an Oly OM-D, because DxOMark says it's better than a 7D anyway, right? : ) . Personally, I don't care what gear you use.

 

[Canihaspicture]

Then we go into the "shadows" and there I select an area and let photoshop with auto contrast calculate pixel values​​, and if there are any banding, noise.This is the difference in the lower level = DR
blah blah

So you're saying if as a photographer I constantly screw up exposure I should get a Nikon? With the 5dMk III if I come anywhere near correct exposure within a few stops (which is a huge range) then I get a great photo with little noise if any. With the Nikon even if I absolutely nail exposure I'll get noise at higher ISOs without changing shadows AT ALL...

Correctly expose your shots in camera and get back to us about noise.

 

[MarkII]

Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

Insulting the original poster while being wrong yourself is not particularly smart.

Of course you can achieve more than the per-pixel DR by downsampling. Suppose two cameras have identical idealised pixel-level DR, but one has double the resolution. With even lighting, the light-per-area is the same on each sensor, but one sensor has double the number of pixels and hence double the headroom. Of course, the higher resolution sensor has more shadow noise, but down-sampling reduces the noise and hence the result is greater DR - more than can be achieved by a single pixel.

In the real world, the light is probably not equally distributed between adjacent pixels, meaning that one may clip before the other (you want something useful from the resolution increase, after-all), and things like system noise also complicate things. However, none of this prevents a down-sampled image having more per-pixel DR than the original sensor pixels, particularly when the downsampling factor is large (4.5x for the D800).

The DXOMark tests are lab-tests, and probably use uniform images for DR (which will tend to maximise the DR benefit from downsampling a D800) and use a lower-noise floor limit that may or may properly account from the pattern noise on many Canon sensors. The DXOMark tests appear to be well performed and accurate - though you need to understand what they are measuring.

If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance. A good example is the 1-bit DAC in some CD players. The DAC is produces a single-bit signal that is hugely oversampled (in the time domain). This is then down-sampled by a filter, yielding a result equivalent to a 16 bit linear DAC (or more).

So not only is it possible to increase DR beyond the native capture range, it is a widely used technique in signal processing.

Of course, the real problem here is that people only want the numbers to show that their purchase is somehow the best and opinion is as valid as understanding. I have seen similar discussions at school in the 1980's - my home computer is better than yours. It is all a bit depressing really.

 

[thepancakeman]

All these posts that say of course you can have greater range on outputs than the sensor is capable of:

On a scale of 1-10, I give you an 11!

<please note the obvious logic flaw and sarcasm>

"But this one goes to 11!"

Or maybe this is a case of "lies, damn lies, and statistics"? I dunno, it just seems be to be missing the forest for the trees.

 

[PackLight]

now this was more a answer to the Neuro who finds it difficult to understand that DXO measurements also can be seen / reflected in an image.

Sorry, but what the hell gives you the idea that I don't understand that what DxO measures can be made to show up in images. Please try to comprehend the following:

• There are differences between Canon and Nikon sensors
  
  
• Some aspects of Nikon sensor performance are superior to Canon sensor performance
• Some aspects of Canon sensor performance are superior to Nikon sensor performance
• DxOMark attempts to quantify those differences and defines their own arbitraty priorities to reduce those measurements to 'Scores'
• DxOMark Scores are relevant only in specific situations, and those situations may or may not have any relevance for the user of a camera
• DxOMark Scores attempt to represent a complex system with a single numeric comparison, an effort which is basically doomed to failure from the start, and is ultimately meaningless

So, if you routinely find yourself in situations where you must push shadow exposures 4 stops with less noise impact, you should probably be shooting Nikon (or asking how you manage to get yourself into those situations in the first place..). If you want to handhold your 600mm lens, you should probably be shooting Canon (or use a 300mm lens with an Oly OM-D, because DxOMark says it's better than a 7D anyway, right? : ) . Personally, I don't care what gear you use.

give one example there Canons sensor is better

Here you go;

The Canon sensor can fit in the D body with its excelent auto focus and keeper rate,
where as the D800 fits inside a body that has a marginal AF keeper rate.

The Canon sensor can use both the excelent line up of Canon lenses, and with adapter the Nikons as well,
Where the Nikons can not use the Canon lenses.

But, I guess you were talking naked sensors?

 

[elflord]

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask. ;D

Well, for a start, they run what is by any reasonable measure the leading sensor benchmark site.

They also implemented raw processing software.

How many industry leading benchmarks and raw converters do their critics on camera rumors sites publish ?

 

[elflord]

All these posts that say of course you can have greater range on outputs than the sensor is capable of:

On a scale of 1-10, I give you an 11!

<please note the obvious logic flaw and sarcasm>

"But this one goes to 11!"

Or maybe this is a case of "lies, damn lies, and statistics"? I dunno, it just seems be to be missing the forest for the trees.

In fact quite the opposite, it is you who are missing the forest for the trees, or in this case, not comprehending that "the sensor" does not consist of a single pixel.