DxOMark vs. Reality

[Zlatko]

So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.

Then photographers would be correct to ignore the Overall Score.

 

[elflord]

Then photographers would be correct to ignore the Overall Score.

If you care about sensor performance, you should pay attention to DxO's measurements because no-one else is going to do a better job at benchmarking sensor performance. Not the overall score, but the use case scores and the graphs.

 

[elflord]

There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous? Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity. But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

:

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.

 

[Zlatko]

There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous? Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity. But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

:

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.

What is analogous is that his example condenses everything into a single number, which you agree is problematic.

 

[Loswr]

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. But how many people look at the Measurements? I suspect it's much more common for people to just stop at the scores. Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements. They report DR, for example, but there's no mention that it only applies at base ISO. That original DxO bias is propagated elsewhere.

 

[Northstar]

They report DR, for example, but there's no mention that it only applies at base ISO. That original DxO bias is propagated elsewhere.

Wow, I never knew this. I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number? I don't understand why you wouldn't do it as an average across the ISO range? Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

 

[elflord]

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless.

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

To me what seems to be driving the outrage here is that Canon sensors get lower DxO scores than Sony sensors. But the relative rankings here are pretty well deserved -- Canon gets trounced at base ISO. It's not even close. At high ISOs, sometimes the Canon sensor inches ahead but the gap is small. So while it's true that there are many shooting scenarios where the Canon sensor is equal or even slightly better than the Sony, if you had to honestly answer the question, who makes the better sensor and had to give a straightforward answer without going into the nuances of use cases, could you say "Canon" with a straight face ?

But how many people look at the Measurements? I suspect it's much more common for people to just stop at the scores.

This is probably true, and if they are just looking at the scores, they are apparently not interested in digging any deeper than that.

Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements. They report DR, for example, but there's no mention that it only applies at base ISO. That original DxO bias is propagated elsewhere.

I don't completely agree with this. The "high ISO score" already takes into account performance at higher ISOs. This score is a pretty good indicator of performance for those who do most of their shooting above base ISO. The dynamic range score is a pretty good indicator if someone is shooting at base ISO.

 

[Loswr]

They report DR, for example, but there's no mention that it only applies at base ISO. That original DxO bias is propagated elsewhere.

Wow, I never knew this. I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number? I don't understand why you wouldn't do it as an average across the ISO range? Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

They call it a 'Landscape Score' so I guess the logic is that you're shooting on a tripod using base ISO.

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless.

Just doesn't follow. The score does not need to be "universally applicable" to be useful.

I'll acknowledge that there is probably no such thing as 'universally applicable'. But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual. If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod? How does DR at base ISO help me evaluate a sensor for my needs? The measurement is useful, the Landscape Score is not.

 

[elflord]

Wow, I never knew this. I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?

No, they don't do this (and it would be a terrible idea). They do take several measurements, but the published use case number comes from base ISO. Never a good idea to rely on guesswork to determine how they got the number.

I don't understand why you wouldn't do it as an average across the ISO range? Especially when you consider that most photographers outside of a studio use the full ISO range.

The use case score is called "landscape" suggesting stationary subject matter that is typically shot at base ISO. Landscape photographers have tripods.

There is a high ISO use case score which measures how well image quality holds up when ISO is cranked up (it basically determines the highest ISO setting for which predetermined SNR and dynamic range criteria are met).

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

 

[Zlatko]

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers? Oh well. If the writer on Luminous Landscape is correct, then the Overall Score is "biased toward single shot HDR at low ISO settings" — a capability we never had in the past and which we may infrequently need. If the Overall Score has to have a bias, that doesn't even sound like the correct bias to have. Likewise, the Portrait Score essentially measures "chroma noise in the dark parts of a low-ISO image" — not exactly the measure of quality when we choose a camera for portrait photography.

 

[elflord]

I'll acknowledge that there is probably no such thing as 'universally applicable'. But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual. If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

That's true, but it's not the same thing as saying that there do not exist individuals for whom it's useful. The user for whom such a blunt instrument is useful is not even going to go to the trouble of addressing whether or not the score addresses their use case(s).

It would be better if the aggregate scores were disclosed, though if they can be reverse engineered within a point or two I don't see this as a major issue (again consumers of this aggregate score would not pay attention to the formula if it were published anyway)

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod? How does DR at base ISO help me evaluate a sensor for my needs? The measurement is useful, the Landscape Score is not.

I don't see how coming up with a benchmark for every conceivable use case (or just not publishing use case benchmarks at all, thus forcing everyone to go through the measurements) would make things any more clear or less confusing.

So it's true that some use cases might require more attention to detail on part of the user. The good news is that DxO make it all available.

 

[elflord]

I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

Photographers "should" pay attention to the actual measurements and properly understand sensor benchmarking.

However, not all "buyers of cameras" are serious photographers, and not all of them are as attentive to the nuances of sensor benchmarking as photographers should be.

Reviewing the raw measurements is better than reviewing the use case scores, which is better than reviewing the aggregate score, which is better than not reviewing any sensor benchmark.

 

[Zlatko]

I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

I think photographers should look at the measurements when they measure something of relevance to the photographer, and ignore the Overall Score.

 

[Kernuak]

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.

 

[leolol]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.

 

[Kernuak]

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.

I try to follow you but do not understand what you mean.
please explain

In low light conditions, the dynamic range in a scene is much less than any modern sensor can capture, therefore whether or not one sensor can capture more DR than another is irrelevant. As I've never measured it, it is rather subjective, but I would estimate that DR is in the 6 stop range for many of the low light conditions that pushed me into buying the 5D MkIII (that's purely an example and wasn't the only reason for buying it). When you start getting into that sort of lighting, then a test should be looking more at the amount of noise in an image and less at how much DR the sensor can capture. DR is desirable for some photography, but unimportant in others and any test of a sensor should reflect that in any objective testing. What I would like to know, is how DxO come up with their cut-off points. For example, why 9 stops of DR for low light testing? Why not 10 or 8? Like Neuro and one or two others, I have been involved in scientific research (albeit nothing to do with optics) and I know that statistics can be used to emphasise pretty much anything if you don't look too closely. That's why it is important to do the right tests and the right analysis. You get it wrong and don't apply proper rigor to the testing in medical research and you could put patients' lives at risk further down the line.
Ultimately though, photography is an art form and therefore subjective. What one person might find acceptable, someone else might find it objectionable. Nowhere is this more obvious than noise and some even add noise to add atmosphere. Even blown highlights or blocked out shadows can add to an image if done correctly, although again, some people will hate it while others love it.

 

[Kernuak]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.

Actually, I don't think he was saying any camera or sensor was better than another, rather that the perception of "better" (however it is measured) is actually irrelevant. The only thing that matters for a business is sales figures. If sales or profits are increasing, then they are achieving their goals (at least in most cases).

 

[Loswr]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

No, I base my conclusions on which camera is better (for me) using criteria and performance aspects that matter to me. In my opinion, that's what everyone buying a dSLR should use to make their own decision, and in most cases, that's likely the case.

It is a fact that Canon sells more dSLRs than Nikon.

Takes together, that suggests that in aggregate, more people have judged that Canon dSLRs satisfy more of their decision-making criteria.

Consider - there's an election, at the end of which (hanging chads notwithstanding), there's a winning candidate and a losing candidate. 'Winner' is based on a count of votes. You voted for the candidate you thought was 'the best'...but that candidate may not have won. You're entitled to your opinion, and if you picked the losing candidate, that doesn't invalidate your opinion. But at the end of the counting, you need to understand that the candidate you liked best lost, and your opinion is in the minority.

There are lots of ways to judge something. The problem is that the vast majority of them are biased by personal opinion. What's best for you isn't necessarily best for everyone. I use sales figures as a metric because that's one of very few unbiased metrics available. We could argue all day about which candidate would be 'best for the country' or which camera is 'the best for taking pictures'. But there was an election, with a winner and a loser...and Canon has sold more cameras than Nikon.

Those are the facts, plain and simple. If it helps, while you don't have the option to be governed/represented by the losing candidate, you can go buy a Nikon dSLR.

 

[tortilla]

A skilled photographer is not the same as he or she is a good copyist and know how a digital work flou works. Have you anytime worked in a darkroom with a negative film then you will understand what I mean and likewise it is with a raw files.

A skilled photographer with also skills in the digital "dark room" kan do so much more than a photographer with no photoshops skills. And even more with a camera who has 14 stops DR.
best regards , pro since 30 years back.

Well, when you need 14 stops DR to "rescue" your pictures via Photoshop, then you should ask yourself, if you`re a real pro.
When it comes to the the point where you really need 14 stops of DR, you've taken your picture in a wrong way.

And: none will see the difference. Neither a customer nor other pro photographer.

So, does that mean HDR imaging is only for amateurs like me who are not able to exposure right, because they don't find the full-automatic setting? Real world lighting situations can sometimes have way above 14 EV DR.

 

[Loswr]

I prefer to look at trends rather than specific points in time, i.e I'm more interested in knowing if Canon/Nikon's DSLR share is growing/shrinking than what specific percentage it is.

That's fine, as long as you realize that one quarter does not equal a trend... FWIW, Canon has gained market share at Nikon's expense for the past 5 years or so.