DxO Mark

[wockawocka]

When it comes to looking at an image I've shot with a camera I've used, studied the raw files of etc I've found the DXOmark scores completely out of whack.

Completely and I wouldn't trust anything on there to relate to real world organic use.

 

[Woody]

Excuse me, but what I don't understand is why there is so much discussion regarding DxO Mark?...

It should also be clear that marketing statements ("image science"?) are sometimes contrary to actual fact. So please excuse DxO for publishing what managers think they should publish, because as a company they have any right to do so. On the other hand there is no reason to complain about this or that score being somewhat different from other tests or actual real life experience.

There is a lot of discussion because many people take them seriously, including somewhat respectable sites like DPReview. Luckily the latter carry out their own tests too.

 

[seekthedragon]

PS: I really would like to hear more about those notable examples.

Here's one obvious one that I've run across:

When DxOMark first publised their measurements and review of the EF 70-200mm f/2.8L IS II, they stated that the MkI version of the 70-200/2.8L IS was sharper. They stated that the, "The Canon EF 70-200mm f/2.8L IS II USM offers slightly less resolution with 51 lp/mm compared to the excellent 61 lp/mm of the Canon EF 70-200mm f/2.8L IS USM," and went on to explain that it was due to 'less homogenous behavior across the field'. They concluded that, "...the overall scores come out slightly in favor of the previous version of the Canon EF 70-200mm f/2.8L IS USM, especially for Travel and Sport photography, which are the main use cases of these telephoto zoom lenses."

That conclusion differed from pretty much every other reviewer/tester who compared the two versions of the lens, and found that the MkII was hands-down the better lens (the f/2.8L IS MkI was considered to be a worse performer optically than both the 70-200/4L IS and the 70-200/2.8L non-IS). Comments can be added to their review pages, and several people commented that from personal experience with both lenses and reviews on DPR, PZ, et al., had all found that the MkII was the better lens. A DxO Labs employee replied, "Thanks for bringing this potential mistake to our attention. But, after checking with all our experts in the lab, there isn’t really a mistake...overall the Mark 1 has a slightly higher and more homogeneous resolution. So, it scores better on a full frame camera, like the Canon 5D Mark II used in the review."

The above quotes are copied from the review page for the 70-200 II, which has not been edited. But, if you compare these two lenses on the 5DII today (screenshot below), you can see that under their new P-Mpix measure for sharpness, the MkII version of the lens performs better than the MkI. They also updated their 'use case scores' so that the MkII is now rated higher for Traval and Sport photography. So…either they re-tested the lens (a different copy, presumably), or whatever black-box factors applied to convert real units (lp/mm) to units that DxO made up (P-Mpix) were applied in a nonlinear manner that somehow favored the MkII version of the lens. I suspect the former is the case, but they said nothing about it, did not add any sort of notation to the original review (which would have been appropriate), or do anything else to acknowledge their mistake.

DxO has used the tagline 'Image Science'. Speaking as a card-carrying scientist, when we discover a mistake in previously published data (it happens), we inform the journal and they publish a correction. DxO's failure to do so in this case is what I'd call 'Bad Science'.

This is quite serious. Thanks for sharing.

 

[Rienzphotoz]

PS: I really would like to hear more about those notable examples.

Here's one obvious one that I've run across:

When DxOMark first publised their measurements and review of the EF 70-200mm f/2.8L IS II, they stated that the MkI version of the 70-200/2.8L IS was sharper. They stated that the, "The Canon EF 70-200mm f/2.8L IS II USM offers slightly less resolution with 51 lp/mm compared to the excellent 61 lp/mm of the Canon EF 70-200mm f/2.8L IS USM," and went on to explain that it was due to 'less homogenous behavior across the field'. They concluded that, "...the overall scores come out slightly in favor of the previous version of the Canon EF 70-200mm f/2.8L IS USM, especially for Travel and Sport photography, which are the main use cases of these telephoto zoom lenses."

That conclusion differed from pretty much every other reviewer/tester who compared the two versions of the lens, and found that the MkII was hands-down the better lens (the f/2.8L IS MkI was considered to be a worse performer optically than both the 70-200/4L IS and the 70-200/2.8L non-IS). Comments can be added to their review pages, and several people commented that from personal experience with both lenses and reviews on DPR, PZ, et al., had all found that the MkII was the better lens. A DxO Labs employee replied, "Thanks for bringing this potential mistake to our attention. But, after checking with all our experts in the lab, there isn’t really a mistake...overall the Mark 1 has a slightly higher and more homogeneous resolution. So, it scores better on a full frame camera, like the Canon 5D Mark II used in the review."

The above quotes are copied from the review page for the 70-200 II, which has not been edited. But, if you compare these two lenses on the 5DII today (screenshot below), you can see that under their new P-Mpix measure for sharpness, the MkII version of the lens performs better than the MkI. They also updated their 'use case scores' so that the MkII is now rated higher for Traval and Sport photography. So…either they re-tested the lens (a different copy, presumably), or whatever black-box factors applied to convert real units (lp/mm) to units that DxO made up (P-Mpix) were applied in a nonlinear manner that somehow favored the MkII version of the lens. I suspect the former is the case, but they said nothing about it, did not add any sort of notation to the original review (which would have been appropriate), or do anything else to acknowledge their mistake.

DxO has used the tagline 'Image Science'. Speaking as a card-carrying scientist, when we discover a mistake in previously published data (it happens), we inform the journal and they publish a correction. DxO's failure to do so in this case is what I'd call 'Bad Science'.

NO! ... that cannot be ... DxO can NEVER be wrong, they are the god of camera gear tests ;D No I refuse to believe this false information ;D

 

[mackguyver]

Neuro's example was the main one I was talking about and it's pretty sad they didn't publish a retraction. There was another review where they just changed the text, but I can't remember which one it was.

I wouldn't say their data is worthless, however. The scores, yes, the analysis, most times, but their measurements are pretty reliable. I've found the measurements on sharpness, CA, distortion, and vignetting to match my lens with the exception of my 180mm f/3.5L, which is extremely sharp in my experience, vs. their measurements. My guess is that they measured it a telephoto distance, where it has average performance, instead of at macro distance, where it is superb.

If you use the tools yourself, and compare lens metrics, not scores, it's a great and FREE resource.