September 16, 2014, 09:51:26 PM

Author Topic: DxOMark vs. Reality  (Read 79119 times)

Maui5150

  • 7D
  • *****
  • Posts: 376
    • View Profile
Re: DxOMark vs. Reality
« Reply #165 on: December 19, 2012, 09:23:24 AM »
DxOMark scores are junk and meaningless.  While there may be a "methodology" to their testing, their score are meaningless for comparison between two cameras, and this is not a Nikon versus Canon debate, but when you see clearly more advanced and better performing camera producing lower results, that alone is justification that the significance of their numbers is in fact, insignificant. 

Is the D600 5 points better than the D4? 

According to dxOMarks, the D600 is a 94 and one of the best cameras tested, and the D4 a lousy 89. 

By DxOMarks scores, the Nikon flagship is crap and over priced.

As well... how can you have a score of "94" and then try shooting above ISO 3200 and see what you get for results. 

If money was not an object, would you buy a D4 or a D600?    According to Dx0marks you are an idiot if you buy the D4 because the D600 is vastly a better scoring camera...

Then again... I think the real idiots are the one who pay attention to Dx0Marks.


canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #165 on: December 19, 2012, 09:23:24 AM »

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2890
    • View Profile
Re: DxOMark vs. Reality
« Reply #166 on: December 19, 2012, 09:31:52 AM »

If money was not an object, would you buy a D4 or a D600?    According to Dx0marks you are an idiot if you buy the D4 because the D600 is vastly a better scoring camera...

Well I can definitely say that I wouldn't buy a D4 - or a 1DX. They're just too big and bulky!

But I think you're mistaking DxO for rating cameras when what they do is rate sensors.

It's up to each individual how important the sensor rating is to their decision making about purchasing a digital camera.

verysimplejason

  • 1D X
  • *******
  • Posts: 1345
    • View Profile
    • My Flickr Account
Re: DxOMark vs. Reality
« Reply #167 on: December 19, 2012, 10:20:58 AM »
So what's your point Mikael?  Did we learn anything new again?  We already know those figures and we've seen it already.  So with those figures, you want to drill us that we should switch to Nikon because it's better? Here we go again...   :)  Reality is, there's more to a camera than a sensor.  Just like there's more to a woman than a pretty face.  If we choose Canon, does that mean it is wrong?  It has been long ago since we had accepted that Sony's sensors are better than Canon but what's the point?  Most of us made our decision because we considered the camera as a whole and we find Canon more attractive than Nikon.  Hmmm... Are you sure you're not a marketing agent of Nikon?  :D

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14381
    • View Profile
Re: DxOMark vs. Reality
« Reply #168 on: December 19, 2012, 10:30:49 AM »
DxOMark scores are junk and meaningless.
Please point why and  how the DXO sensor measurement is faulty.
Note that Maui150 stated 'scores', not 'measurements'.  IMO, their sensor Measurements are fine, it's their Sensor Scores, that are faulty. 

First, those scores are derived from the measurements in an incompletely disclosed manner - it's a 'weighted average' but weighted how? DxO themselves have stated, "The DxOMark Camera Sensor score is under normal conditions a weighted average of noise, dynamic range and color sensitivity information. But some non-linearities are deliberately included in the algorithm to avoid a clear weakness in one area from being masked by a strength in other areas," (source).  An analogy might be the Dow Jones Industrial Average, which is a price-weighted index - what if the Dow decided to give five of those 30 companies greater weight in the index, but didn't tell us which companies were the chosen five, or whether it was the same five from day to day?  If they did that, the DJIA would have little utility as an index, although we'd still know the closing share prices of the 30 index companies, much like the DxOMark Scores have little utility as a sensor benchmark, although we know the results of the individual DxOMark measurements.

Second, the Overall Score is biased toward performance at base ISO.  Two of the three subscores (Landscape and Portrait) consider performance only at base ISO, rather than considering performance of those metrics across the range of available ISOs for the sensor being tested.  Not all amplifiers are created equal, and DxO's measurements clearly show that when comparing two sensors, while one may have better DR and color depth at base ISO (e.g. 100), the other may have better DR and color depth at ISO 3200.  However, only base ISO contributes to those subscores.  That's a bias in the subscores, and thus, it's carried forward into the overall score.

Now...put those two together.  An Overall Score derived from three subscores with two considering only base ISO, meaning a 2:1 bias in favor of base ISO performance.  An Overall Score which is a weighted average of three subscores with unknown weightings, affected by intentionally-selected but nondisclosed nonlinearities.  Do the weightings and undisclosed skewings of the algorithm correct for that potential 2:1 bias, or make it worse?  We don't know. 

So, while it's possible to look at the specific measurements in their data (and I applaud DxO for publishing those data), and while it's possible to post those measurement data over and over again, it doesn't change the fact that DxOMark's Overall Score and Use-Case Scores are derived from those data in an ambiguous and undisclosed manner, and that makes them faulty.   
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

verysimplejason

  • 1D X
  • *******
  • Posts: 1345
    • View Profile
    • My Flickr Account
Re: DxOMark vs. Reality
« Reply #169 on: December 19, 2012, 10:39:43 AM »
My point is, why troing garbage at DXO? noting else

I respect the scores of DXO but I take them with a grain of salt.  As Neuro pointed out, if you (DXO) really want to give measurements/scores you need to divulge everything and take everything into account (no bias, take equally everything into consideration).  Without doing that, you'll never be able to escape scrutiny.

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #170 on: December 19, 2012, 11:03:37 AM »
DXO seems for all intents to do exactly what they claim to do: take precise, repeatable measurements of certain aspects of digital camera sensors, and condense these measurements into continually more and more abstract performance figures that are easier to read and remember but harder to relate to physics. This is similar to how a camera review site might give a camera a score of 67 out of 100 points - except that in that case the reviewers subjective opinion tends to be a factor.


I think the point is the "abstract performance figures".
I think you hit the point, a camera review site may gvie a score, but when DxO decides how to create their "abstract performance figures" they inject their subjective opinion in to how the "abstract performance figures" are presented. This is the flaw in their method. They are doing the same thing a camera  (sensor)review site would do.


neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14381
    • View Profile
Re: DxOMark vs. Reality
« Reply #171 on: December 19, 2012, 11:10:56 AM »
Peter: So how fair is the DxOMark Camera Sensor Score?
There is a difficult but important question. Probably every image quality scientist[74] in the world would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws.

...then he goes on to discuss the low ISO bias which I mentioned, and he argues that DxOMark should release their methods for determining the overall score, the two points which I find most objectionable.  He also lists several other issues with DxOMark's process.

As he states, "If you compare the DxOMark data in Figure 7 for a number of prominent cameras you would get a more balanced impression about which camera to buy than by just looking at the overall DxOMark Sensor score. If you focus on the latter, you would strongly prefer the Nikon D800 with its excellent low ISO dynamic range. But this emphasizes one aspect of the sensor (essentially the ability to do single shot HDR) that provides a capability we never had in the past. It is a feature which we may infrequently need – and one that some types of users may never see (e.g. if you shoot JPG). However, at sufficiently high ISO, other models win. High ISO usage may be a more relevant usage for many users than HDR ability at low ISO."

That illustrates the issue with using DxOMark's Scores as a benchmark for sensors, and emphasizes the importance of examining the actual measurements (sort of like reading just the abstract of a scientific publication - you are relying on the author's conclusions drawn from their own data interpreted through the lens of their own biases, when it's preferable to examine the data for yourself and draw your own conclusions...which may or may not agree with those of the author).
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #171 on: December 19, 2012, 11:10:56 AM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14381
    • View Profile
Re: DxOMark vs. Reality
« Reply #172 on: December 19, 2012, 11:30:23 AM »
well , for me there are no problems, Im looking  for example at FWC, QE, read out noise , and then get  an idea of the sensors behavior, and then I look at the practical difference, DR, noise, banding etc by comparing the cameras and under the exact same parameters.
Which I have done 100 of times

Exactly.  But you are clearly approaching the issue of sensor performance with a much broader and deeper knowledge base than a typical consumer.  I would bet that most people who look at DxOMark comparisons have no idea what the abbreviations FWC and QE even stand for, much less what those concepts actually mean in terms of sensor performance.  Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions. 
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

thepancakeman

  • 6D
  • *****
  • Posts: 457
  • If at first you don't succeed, don't try skydiving
    • View Profile
Re: DxOMark vs. Reality
« Reply #173 on: December 19, 2012, 11:41:41 AM »
Under the "cameras" tab on their page, I see "camera sensor database", "camera sensor ratings" and "compare camera sensors".

I don't think they're making it a secret that they are benchmarking sensors.

Not secret, just enough under the radar that it's easily overlooked by average consumers.

Quote
Quote
They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities.  Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria.

What descriptions would be more appropriate in your opinion ?

Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

Quote
Quote
I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score.  Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.

How precisely are they trying to "mislead" people ? This is a step beyond simply criticising their choice of naming, here you appear to be accusing them of choosing the names in such a way as to favour their business.

Well, please do substantiate this accusation a little. For example, how do they benefit from calling their high ISO use case "Sports" instead of "event photography" or "wedding photography" ?

I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon.  I don't know this for a fact or why they would do such; maybe they own stock.  But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports.  Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)
« Last Edit: December 19, 2012, 11:57:14 AM by thepancakeman »

MarkII

  • PowerShot G1 X II
  • ***
  • Posts: 65
    • View Profile
Re: DxOMark vs. Reality
« Reply #174 on: December 19, 2012, 12:52:47 PM »
Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions.
To be fair, that is not really DXO's problem. The internet is stuffed full of sites that either accidentally or deliberately pick and choose numbers that give a biased impression.  Snapsort offends in many more grievous ways than their use of sensor scores.

And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions. There is so much information to digest, they would never get to the point of consuming anything. Though they do seem to get upset if anything suggests that their decision was less than perfect...

Maybe DXO should add thumbs-up and thumbs-down buttons. Click the button to show what you think about the score. By storing the information in a browser cookie, they could adjust their scoring metrics to suit each individiual taste :-)

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14381
    • View Profile
Re: DxOMark vs. Reality
« Reply #175 on: December 19, 2012, 12:58:09 PM »
And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people.  But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

thepancakeman

  • 6D
  • *****
  • Posts: 457
  • If at first you don't succeed, don't try skydiving
    • View Profile
Re: DxOMark vs. Reality
« Reply #176 on: December 19, 2012, 02:09:56 PM »
And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people.  But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.

Ah, but DXOMark doesn't just do dSLR's.   ;)

Maui5150

  • 7D
  • *****
  • Posts: 376
    • View Profile
Re: DxOMark vs. Reality
« Reply #177 on: December 19, 2012, 03:02:23 PM »

your answer is  clever like  as -  all Canons gear is worthless
Please point why and  how the DXO sensor measurement is faulty.

Which is a better overall camera?  The D4 or the D600?

Which will take the best picture?  The D4 or the D600?

Which has a better Dx0Mark Score? 

The answer to the first 2 is the D4, and any HONEST person would agree

Yet the Dx0 Mark for the D600 is over 5% higher? 

Sort of PROVES my point the the Dx0Mark score is MEANINGLESS

Or perhaps you are trying to say that Nikon throws poor sensors in their flagship camera (meaning that the D4 is SUPER over priced)

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #177 on: December 19, 2012, 03:02:23 PM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #178 on: December 19, 2012, 06:55:21 PM »
look up my recent hisotory on posting about DXO...i just don't believe they have their "act" together.

I did. It looks like you've gone on a fishing expedition to find some kind of perceived wrongdoing.  I suppose this is consistent with casting aspersions and not being forthright about what your true objections are. One recent post was a complaint about typos, the other about their lens benchmarking. None of this has any relevance to or casts any doubt on their status as the industry leader in sensor benchmarking.

It seems to me that you didn't like it that Canon cameras did not score well on their benchmarks and decided to smear them on that basis.

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #179 on: December 19, 2012, 06:58:57 PM »
Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

But they use those labels as well.

Quote
I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon.  I don't know this for a fact or why they would do such; maybe they own stock.  But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports.  Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)

Well, they don't benchmark AF systems.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #179 on: December 19, 2012, 06:58:57 PM »