DxOMark vs. Reality

[Northstar]

hey ELF...i already implied why....read again the last part of my post.

as I've already stated on this forum...it's of my opinion, that DXO is a group of lazy and unorganized....enough said.

I do understand that you're casting aspersions as opposed to stating your views in a more forthright manner (which might create an expectation that you substantiate your allegations).

What you do not understand is that I'm asking you to put up or shut up. If you have some insight or knowledge that supports your view, please do not be shy -- share it with us. If you don't have any such insight or knowledge, it does raise some questions as to why you are prone to make unsubstantiated and unprovoked attacks on the reputation and character of DxO.

look up my recent hisotory on posting about DXO...i just don't believe they have their "act" together.

 

[neuroanatomist]

What you do not understand is that I'm asking you to put up or shut up.

That has been asked of others on this thread, with no 'putting up' evident.

you have the right to think what you want.
we are some who do not share your opinion, you must also accept that

+1 Without supporting evidence, statements are merely opinion, and as you say, we are all entitled to our own with no expectation that others will agree.

Just to remind people of the central thesis of this thread - it's not that DxOMark scores are 'cooked' or 'false', nor even that the scores are biased in favor one brand over another. I've seen no evidence of the former (quite the opposite), and while the latter may be true, there's no evidence it's driven by anything other than logic - we generally try to shoot at the lowest ISO possible, so there's some logic to biasing the scores to base ISO.

Rather, the central point is that DxOMark's scores seem to have had no meaningful impact on the real world aggregate buying decisions of consumers, as demonstrated by Canon's and Nikon's own financial reports detailing their global sales figures. Put succinctly as a statement comprising fact, Nikon dSLRs have beaten Canon's in DxOMark scores for several years, and Canon's dSLRs have outsold Nikon's for those same several years.

 

[PackLight]

make unsubstantiated and unprovoked attacks on the reputation and character of DxO.

Interesting statement;

I think anytime a company presents information to the public, if it is misleading in nature it calls in to question the character of the company.

The tests DXO performs on sensors, if the data is presented for what it is I would not question their presentation or motive. When they take the data and manipulate it to arrive at a score they created with a description that they perceive or make us want to believe is correct, I have to question why they would do this.

How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

 

[Maui5150]

DxOMark scores are junk and meaningless. While there may be a "methodology" to their testing, their score are meaningless for comparison between two cameras, and this is not a Nikon versus Canon debate, but when you see clearly more advanced and better performing camera producing lower results, that alone is justification that the significance of their numbers is in fact, insignificant.

Is the D600 5 points better than the D4?

According to dxOMarks, the D600 is a 94 and one of the best cameras tested, and the D4 a lousy 89.

By DxOMarks scores, the Nikon flagship is crap and over priced.

As well... how can you have a score of "94" and then try shooting above ISO 3200 and see what you get for results.

If money was not an object, would you buy a D4 or a D600? According to Dx0marks you are an idiot if you buy the D4 because the D600 is vastly a better scoring camera...

Then again... I think the real idiots are the one who pay attention to Dx0Marks.

 

[verysimplejason]

So what's your point Mikael? Did we learn anything new again? We already know those figures and we've seen it already. So with those figures, you want to drill us that we should switch to Nikon because it's better? Here we go again... Reality is, there's more to a camera than a sensor. Just like there's more to a woman than a pretty face. If we choose Canon, does that mean it is wrong? It has been long ago since we had accepted that Sony's sensors are better than Canon but what's the point? Most of us made our decision because we considered the camera as a whole and we find Canon more attractive than Nikon. Hmmm... Are you sure you're not a marketing agent of Nikon?

 

[neuroanatomist]

DxOMark scores are junk and meaningless.

Please point why and how the DXO sensor measurement is faulty.

Note that Maui150 stated 'scores', not 'measurements'. IMO, their sensor Measurements are fine, it's their Sensor Scores, that are faulty.

First, those scores are derived from the measurements in an incompletely disclosed manner - it's a 'weighted average' but weighted how? DxO themselves have stated, "The DxOMark Camera Sensor score is under normal conditions a weighted average of noise, dynamic range and color sensitivity information. But some non-linearities are deliberately included in the algorithm to avoid a clear weakness in one area from being masked by a strength in other areas," (source). An analogy might be the Dow Jones Industrial Average, which is a price-weighted index - what if the Dow decided to give five of those 30 companies greater weight in the index, but didn't tell us which companies were the chosen five, or whether it was the same five from day to day? If they did that, the DJIA would have little utility as an index, although we'd still know the closing share prices of the 30 index companies, much like the DxOMark Scores have little utility as a sensor benchmark, although we know the results of the individual DxOMark measurements.

Second, the Overall Score is biased toward performance at base ISO. Two of the three subscores (Landscape and Portrait) consider performance only at base ISO, rather than considering performance of those metrics across the range of available ISOs for the sensor being tested. Not all amplifiers are created equal, and DxO's measurements clearly show that when comparing two sensors, while one may have better DR and color depth at base ISO (e.g. 100), the other may have better DR and color depth at ISO 3200. However, only base ISO contributes to those subscores. That's a bias in the subscores, and thus, it's carried forward into the overall score.

Now...put those two together. An Overall Score derived from three subscores with two considering only base ISO, meaning a 2:1 bias in favor of base ISO performance. An Overall Score which is a weighted average of three subscores with unknown weightings, affected by intentionally-selected but nondisclosed nonlinearities. Do the weightings and undisclosed skewings of the algorithm correct for that potential 2:1 bias, or make it worse? We don't know.

So, while it's possible to look at the specific measurements in their data (and I applaud DxO for publishing those data), and while it's possible to post those measurement data over and over again, it doesn't change the fact that DxOMark's Overall Score and Use-Case Scores are derived from those data in an ambiguous and undisclosed manner, and that makes them faulty.

 

[verysimplejason]

My point is, why troing garbage at DXO? noting else

I respect the scores of DXO but I take them with a grain of salt. As Neuro pointed out, if you (DXO) really want to give measurements/scores you need to divulge everything and take everything into account (no bias, take equally everything into consideration). Without doing that, you'll never be able to escape scrutiny.

 

[PackLight]

DXO seems for all intents to do exactly what they claim to do: take precise, repeatable measurements of certain aspects of digital camera sensors, and condense these measurements into continually more and more abstract performance figures that are easier to read and remember but harder to relate to physics. This is similar to how a camera review site might give a camera a score of 67 out of 100 points - except that in that case the reviewers subjective opinion tends to be a factor.

I think the point is the "abstract performance figures".
I think you hit the point, a camera review site may gvie a score, but when DxO decides how to create their "abstract performance figures" they inject their subjective opinion in to how the "abstract performance figures" are presented. This is the flaw in their method. They are doing the same thing a camera (sensor)review site would do.

 

[neuroanatomist]

Peter: So how fair is the DxOMark Camera Sensor Score?
There is a difficult but important question. Probably every image quality scientist[74] in the world would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws.

...then he goes on to discuss the low ISO bias which I mentioned, and he argues that DxOMark should release their methods for determining the overall score, the two points which I find most objectionable. He also lists several other issues with DxOMark's process.

As he states, "If you compare the DxOMark data in Figure 7 for a number of prominent cameras you would get a more balanced impression about which camera to buy than by just looking at the overall DxOMark Sensor score. If you focus on the latter, you would strongly prefer the Nikon D800 with its excellent low ISO dynamic range. But this emphasizes one aspect of the sensor (essentially the ability to do single shot HDR) that provides a capability we never had in the past. It is a feature which we may infrequently need – and one that some types of users may never see (e.g. if you shoot JPG). However, at sufficiently high ISO, other models win. High ISO usage may be a more relevant usage for many users than HDR ability at low ISO."

That illustrates the issue with using DxOMark's Scores as a benchmark for sensors, and emphasizes the importance of examining the actual measurements (sort of like reading just the abstract of a scientific publication - you are relying on the author's conclusions drawn from their own data interpreted through the lens of their own biases, when it's preferable to examine the data for yourself and draw your own conclusions...which may or may not agree with those of the author).

 

[neuroanatomist]

well , for me there are no problems, Im looking for example at FWC, QE, read out noise , and then get an idea of the sensors behavior, and then I look at the practical difference, DR, noise, banding etc by comparing the cameras and under the exact same parameters.
Which I have done 100 of times

Exactly. But you are clearly approaching the issue of sensor performance with a much broader and deeper knowledge base than a typical consumer. I would bet that most people who look at DxOMark comparisons have no idea what the abbreviations FWC and QE even stand for, much less what those concepts actually mean in terms of sensor performance. Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions.

 

[thepancakeman]

Under the "cameras" tab on their page, I see "camera sensor database", "camera sensor ratings" and "compare camera sensors".

I don't think they're making it a secret that they are benchmarking sensors.

Not secret, just enough under the radar that it's easily overlooked by average consumers.

They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities. Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria.

What descriptions would be more appropriate in your opinion ?

Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score. Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.

How precisely are they trying to "mislead" people ? This is a step beyond simply criticising their choice of naming, here you appear to be accusing them of choosing the names in such a way as to favour their business.

Well, please do substantiate this accusation a little. For example, how do they benefit from calling their high ISO use case "Sports" instead of "event photography" or "wedding photography" ?

I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon. I don't know this for a fact or why they would do such; maybe they own stock. But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports. Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)

 

[MarkII]

Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions.

To be fair, that is not really DXO's problem. The internet is stuffed full of sites that either accidentally or deliberately pick and choose numbers that give a biased impression. Snapsort offends in many more grievous ways than their use of sensor scores.

And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions. There is so much information to digest, they would never get to the point of consuming anything. Though they do seem to get upset if anything suggests that their decision was less than perfect...

Maybe DXO should add thumbs-up and thumbs-down buttons. Click the button to show what you think about the score. By storing the information in a browser cookie, they could adjust their scoring metrics to suit each individiual taste

 

[neuroanatomist]

And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people. But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.

 

[thepancakeman]

And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people. But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.

Ah, but DXOMark doesn't just do dSLR's.

 

[Maui5150]

your answer is clever like as - all Canons gear is worthless
Please point why and how the DXO sensor measurement is faulty.

Which is a better overall camera? The D4 or the D600?

Which will take the best picture? The D4 or the D600?

Which has a better Dx0Mark Score?

The answer to the first 2 is the D4, and any HONEST person would agree

Yet the Dx0 Mark for the D600 is over 5% higher?

Sort of PROVES my point the the Dx0Mark score is MEANINGLESS

Or perhaps you are trying to say that Nikon throws poor sensors in their flagship camera (meaning that the D4 is SUPER over priced)

 

[elflord]

look up my recent hisotory on posting about DXO...i just don't believe they have their "act" together.

I did. It looks like you've gone on a fishing expedition to find some kind of perceived wrongdoing. I suppose this is consistent with casting aspersions and not being forthright about what your true objections are. One recent post was a complaint about typos, the other about their lens benchmarking. None of this has any relevance to or casts any doubt on their status as the industry leader in sensor benchmarking.

It seems to me that you didn't like it that Canon cameras did not score well on their benchmarks and decided to smear them on that basis.

 

[elflord]

Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

But they use those labels as well.

I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon. I don't know this for a fact or why they would do such; maybe they own stock. But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports. Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)

Well, they don't benchmark AF systems.

 

[elflord]

How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.

 

[elflord]

your answer is clever like as - all Canons gear is worthless
Please point why and how the DXO sensor measurement is faulty.

Which is a better overall camera? The D4 or the D600?

Which will take the best picture? The D4 or the D600?

DxO do not attempt to measure which is "a better overall camera" (whatever that means though I think most would agree that it's the D4) nor do they measure "which will take the best picture" (what does "the best picture" mean ?)

They benchmark sensors.

Now, your answer to the question is ?

 

[verysimplejason]

If we shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow when people are involving other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor? That is why this is titled DXO vs Reality. Reality is what it is. You need to consider everything and not only the sensor. It's elementary my dear Watson...