October 23, 2014, 09:27:29 AM

Author Topic: DxOMark vs. Reality  (Read 81015 times)

Zlatko

  • Guest
Re: DxOMark vs. Reality
« Reply #120 on: December 15, 2012, 01:16:21 PM »
So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.
Then photographers would be correct to ignore the Overall Score.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #120 on: December 15, 2012, 01:16:21 PM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #121 on: December 15, 2012, 01:25:45 PM »
Then photographers would be correct to ignore the Overall Score.

If you care about sensor performance, you should pay attention to DxO's measurements because no-one else is going to do a better job at benchmarking sensor performance. Not the overall score, but the use case scores and the graphs.
« Last Edit: December 15, 2012, 01:28:36 PM by elflord »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #122 on: December 15, 2012, 01:47:36 PM »
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.

Zlatko

  • Guest
Re: DxOMark vs. Reality
« Reply #123 on: December 15, 2012, 02:01:45 PM »
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.
What is analogous is that his example condenses everything into a single number, which you agree is problematic.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14727
    • View Profile
Re: DxOMark vs. Reality
« Reply #124 on: December 15, 2012, 02:03:43 PM »
The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless.  But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores.  Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Northstar

  • 1D X
  • *******
  • Posts: 1484
    • View Profile
Re: DxOMark vs. Reality
« Reply #125 on: December 15, 2012, 02:39:13 PM »
Quote
They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

Sport Shooter

1dX and 5d3... 24-70 2.8ii, 70-200 2.8ii, 1.4xiii and 2xiii, 85, 40mm, 300 2.8L IS....430ex

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #126 on: December 15, 2012, 02:45:36 PM »
The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

To me what seems to be driving the outrage here is that Canon sensors get lower DxO scores than Sony sensors. But the relative rankings here are pretty well deserved -- Canon gets trounced at base ISO. It's not even close. At high ISOs, sometimes the Canon sensor inches ahead but the gap is small. So while it's true that there are many shooting scenarios where the Canon sensor is equal or even slightly better than the Sony, if you had to honestly answer the question, who makes the better sensor  and had to give a straightforward answer without going into the nuances of use cases, could you say "Canon" with a straight face ?

Quote
But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores. 

This is probably true, and if they are just looking at the scores, they are apparently not interested in digging any deeper than that.

Quote
Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

I don't completely agree with this. The "high ISO score" already takes into account performance at higher ISOs. This score is a pretty good indicator of performance for those who do most of their shooting above base ISO. The dynamic range score is a pretty good indicator if someone is shooting at base ISO.


canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #126 on: December 15, 2012, 02:45:36 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14727
    • View Profile
Re: DxOMark vs. Reality
« Reply #127 on: December 15, 2012, 03:03:35 PM »
Quote
They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

They call it a 'Landscape Score' so I guess the logic is that you're shooting on a tripod using base ISO.

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful.

I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not. 
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #128 on: December 15, 2012, 03:06:13 PM »
Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number? 

No, they don't do this (and it would be a terrible idea).  They do take several measurements, but the published use case number comes from base ISO. Never a good idea to rely on guesswork to determine how they got the number.

Quote
I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

The use case score is called "landscape" suggesting stationary subject matter that is typically shot at base ISO. Landscape photographers have tripods.

There is a high ISO use case score which measures how well image quality holds up when ISO is cranked up (it basically determines the highest ISO setting for which predetermined SNR and dynamic range criteria are met).

Quote
75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots


Zlatko

  • Guest
Re: DxOMark vs. Reality
« Reply #129 on: December 15, 2012, 03:23:55 PM »
Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.
I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?  Oh well.  If the writer on Luminous Landscape is correct, then the Overall Score is "biased toward single shot HDR at low ISO settings" — a capability we never had in the past and which we may infrequently need.  If the Overall Score has to have a bias, that doesn't even sound like the correct bias to have.  Likewise, the Portrait Score essentially measures "chroma noise in the dark parts of a low-ISO image" — not exactly the measure of quality when we choose a camera for portrait photography.  :)

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #130 on: December 15, 2012, 03:30:16 PM »
I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

That's true, but it's not the same thing as saying that there do not exist individuals for whom it's useful.  The user for whom such a blunt instrument is useful is not even going to go to the trouble of addressing whether or not the score addresses their use case(s).

It would be better if the aggregate scores were disclosed, though if they can be reverse engineered within a point or two I don't see this as a major issue (again consumers of this aggregate score would not pay attention to the formula if it were published anyway)

Quote
Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not.

I don't see how coming up with a benchmark for every conceivable use case (or just not publishing use case benchmarks at all, thus forcing everyone to go through the measurements)  would make things any more clear or less confusing.

So it's true that some use cases might require more attention to detail on part of the user. The good news is that DxO make it all available.

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #131 on: December 15, 2012, 03:37:58 PM »
I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

Photographers "should" pay attention to the actual measurements and properly understand sensor benchmarking.

However, not all "buyers of cameras" are serious photographers, and not all of them are as attentive to the nuances of sensor benchmarking as photographers should be.

Reviewing the raw measurements is better than reviewing the use case scores, which is better than reviewing the aggregate score, which is better than not reviewing any sensor benchmark.

Zlatko

  • Guest
Re: DxOMark vs. Reality
« Reply #132 on: December 15, 2012, 03:59:00 PM »
I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)
I think photographers should look at the measurements when they measure something of relevance to the photographer, and ignore the Overall Score.
« Last Edit: December 15, 2012, 04:03:09 PM by Zlatko »

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #132 on: December 15, 2012, 03:59:00 PM »

Kernuak

  • 1D X
  • *******
  • Posts: 1108
    • View Profile
    • Avalon Light Photoart
Re: DxOMark vs. Reality
« Reply #133 on: December 15, 2012, 04:00:39 PM »

Quote
75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots
Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.
Canon 5D MkIII, 7D, 300mm L IS f/2.8 and a few other L's

leolol

  • Guest
Re: DxOMark vs. Reality
« Reply #134 on: December 15, 2012, 05:02:13 PM »
@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #134 on: December 15, 2012, 05:02:13 PM »