DxOMark vs. Reality

[Northstar]

I occasionally  read their reviews but take them with a huge grain of salt....they continue to do things that make me think they are unorganized and don't have their "stuff" together.   I have a few other reviewers that I prefer.

The latest thing I noticed was when I checked to see if they had done a review on the 135L, this was in September.  They didn't have a review but they did have a note saying they would be posting a review of the 135L in October.  So I check back in November, no review, but a note saying they'll be reviewing it in November...So I forgot about it until now, I just checked, they still haven't reviewed the lens, and the note has been changed to...(copied from their website)

This product will be tested and reviewed on December on dxomark.com. Stay tuned by subscribing to our newsletter.

So it's December 15th and no review...anyone want to bet that it won't be done in December?  Where I come from, if you say one thing and do another, again, and again....it just isn't good, you lose credibility.

Also, another typo where they write "on December".  Their website is full of typos. 

IMO, they are overrated as a "review" website.

[elflord]

The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data. 

DXOmark are THE industry leader in sensor benchmarking. There is nothing "suspect" about their sensor measurements. The minutiae of sensor benchmarking has been debated here, and the overwhelming conclusion of the discussions is that the measurements for their sensor benchmarks are sound. There is some nitpicking about the way those measurements are aggregated but that's about it.

I personally would hold off buying a camera body until DxO have tested it. I don't have nearly as much confidence in other sources for testing sensors.

Lenses are a separate issue, there are other sources that do a better job of testing lenses.

[Zlatko]

The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data. 

DXOmark are THE industry leader in sensor benchmarking. There is nothing "suspect" about their sensor measurements. The minutiae of sensor benchmarking has been debated here, and the overwhelming conclusion of the discussions is that the measurements for their sensor benchmarks are sound. There is some nitpicking about the way those measurements are aggregated but that's about it.

I personally would hold off buying a camera body until DxO have tested it. I don't have nearly as much confidence in other sources for testing sensors.

According to this article on Luminous Landscape today —
http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml
— the  overall Camera Sensor score is biased toward single shot HDR at low ISO settings — a capability we never had in the past and which we may infrequently need.

So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

We also learn that their Portrait score is misnamed.  "Essentially it measures choma noise in the dark parts of a low-ISO image" — which may be relevant to some photographers, and not at all relevant to others.  This explains why two cameras can have the same Portrait scores and yet one can be clearly better for actual portraits.

[elflord]

So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.

[neuroanatomist]

There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 

[Zlatko]

So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.

Then photographers would be correct to ignore the Overall Score.

[elflord]

Then photographers would be correct to ignore the Overall Score.

If you care about sensor performance, you should pay attention to DxO's measurements because no-one else is going to do a better job at benchmarking sensor performance. Not the overall score, but the use case scores and the graphs.

[elflord]

There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.

[Zlatko]

There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.

What is analogous is that his example condenses everything into a single number, which you agree is problematic.

[neuroanatomist]

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless.  But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores.  Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

[Northstar]

They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

[elflord]

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

To me what seems to be driving the outrage here is that Canon sensors get lower DxO scores than Sony sensors. But the relative rankings here are pretty well deserved -- Canon gets trounced at base ISO. It's not even close. At high ISOs, sometimes the Canon sensor inches ahead but the gap is small. So while it's true that there are many shooting scenarios where the Canon sensor is equal or even slightly better than the Sony, if you had to honestly answer the question, who makes the better sensor  and had to give a straightforward answer without going into the nuances of use cases, could you say "Canon" with a straight face ?

But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores. 

This is probably true, and if they are just looking at the scores, they are apparently not interested in digging any deeper than that.

Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

I don't completely agree with this. The "high ISO score" already takes into account performance at higher ISOs. This score is a pretty good indicator of performance for those who do most of their shooting above base ISO. The dynamic range score is a pretty good indicator if someone is shooting at base ISO.

[neuroanatomist]

They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

They call it a 'Landscape Score' so I guess the logic is that you're shooting on a tripod using base ISO.

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful.

I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not. 

[elflord]

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number? 

No, they don't do this (and it would be a terrible idea).  They do take several measurements, but the published use case number comes from base ISO. Never a good idea to rely on guesswork to determine how they got the number.

I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

The use case score is called "landscape" suggesting stationary subject matter that is typically shot at base ISO. Landscape photographers have tripods.

There is a high ISO use case score which measures how well image quality holds up when ISO is cranked up (it basically determines the highest ISO setting for which predetermined SNR and dynamic range criteria are met).

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

[Zlatko]

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?  Oh well.  If the writer on Luminous Landscape is correct, then the Overall Score is "biased toward single shot HDR at low ISO settings" — a capability we never had in the past and which we may infrequently need.  If the Overall Score has to have a bias, that doesn't even sound like the correct bias to have.  Likewise, the Portrait Score essentially measures "chroma noise in the dark parts of a low-ISO image" — not exactly the measure of quality when we choose a camera for portrait photography.