DxOMark vs. Reality

[neuroanatomist]

If we shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow when people are involving other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor? That is why this is titled DXO vs Reality. Reality is what it is. You need to consider everything and not only the sensor. It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read at DXO what the marks includes and which sensor parameters are measured. http://www.dxomark.com/index.php/About/Sensor-scores

Yep, reality is the whole point. If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs. Has that been happening? Or does Canon still have a greater market share?

 

[verysimplejason]

If we shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow when people are involving other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor? That is why this is titled DXO vs Reality. Reality is what it is. You need to consider everything and not only the sensor. It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read at DXO what the marks includes and which sensor parameters are measured. http://www.dxomark.com/index.php/About/Sensor-scores

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance: nada more.

DXO scores sensors while in REALITY there's more to a camera than a sensor. Is it that hard to understand? That's the problem with DXO and what's almost everybody is pointing out. DXO seems content on grading the sensor but not the whole camera body. That's why this is titled DXOMARK vs REALITY.

 

[TheSuede]

The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

As stated so many times before, you cannot put a "score" on a camera as a complete system for any sort of user average, you have to specify a quite tight definition of the score scope - what it does include and what it doesn't. And it does get tricky when you try to include more than one parameter in a "score".

You may (in my opinion...) for instance put a reasonably real-world relevant score on "low light noise performance". This score will then preferably accurately represent how much noise a camera will show in a shot given an average exposure of exactly "X" lumen seconds per mm2. The measurement "noise per lm*s/mm2" is just a strict definition of what a photographer would call "I want 1/200s at F2.8 in this light".

-But it (the score) will not say anything at all about how well the camera can AF in that same situation. Neither will it say anything about the viewfinder, the fps rate or the color accuracy. The score will be "low light NOISE performance", not "low light performance".
...

But the individual measurement scales and results in themselves that DxO use to build their total "score" are almost beyond reproach. You could possibly fault their presentation mode in some of the results, but not the results in themselves.
The few individuals and organizations that can replicate the accuracy and scientific stringency of their tests all arrive at the same conclusion - there's not much to be said about it except for the fact that DxO are almost 100% repeatable and accurate. The numbers just fit, and anyone measuring the same thing on the same cameras will arrive at the same result. And anyone comparing two cameras out in the real world can - if they adhere to reality in stead of maker-defined bogus ISO values and so on - also visualize exactly what that number means in an image.

So it's up to the reader of the results to make use of the information in a way that he or she can relate to, and here's where most people fall flat on their faces. Unfortunately DxO trip themselves up by trying to apply an overall "score" on a camera, but as far as anyone in the publishing business can tell - you NEED a promise of an end score or other simple graphical or single-digit grading system to get 95% of the readers to even bother starting to read.

If the average readers realizes that they actually might have to think a bit for themselves to get something out of something, they quite often just bugger off somewhere else before even starting to assimilate any real information.

 

[TheSuede]

Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

kind of like saying
:"Everything Apple makes right now has to be crap, they're constantly dropping in market share"
???

Neither of those arguments are worth paying any attention to, since there's more to sales and market shares than measurable product quality.

 

[neuroanatomist]

Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

I could not agree more. This was what I was driving at, initially. At the time I originated this thread, there was a plethora of posts/threads proclaiming the superiority of the D800 (and other Nikon bodies) as cameras because of the DxOMark scores, and nearly as many posts/threads suggesting that Canon was doomed as a corporate entity because of their subpar sensors. The former claim is clearly unsupported by DxOMark, and the latter claim is equally untrue.

Thanks, TheSuede, for your cogent post.

 

[thepancakeman]

If we shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow when people are involving other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor? That is why this is titled DXO vs Reality. Reality is what it is. You need to consider everything and not only the sensor. It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read at DXO what the marks includes and which sensor parameters are measured. http://www.dxomark.com/index.php/About/Sensor-scores

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance: nada more.

The reality is that low-light ISO does not = a good sports camera, but that is how it's being portrayed.

 

[elflord]

Yep, reality is the whole point. If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs. Has that been happening? Or does Canon still have a greater market share?

It is not as the thread title suggests, a "competitor" or an "alternative" to reality but rather a subset thereof.

BTW, if the scores do not substantially affect sales, why all the concern about them "misleading" the "consumer" ?

 

[PackLight]

How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.

Fudging? It wasn't my word.

I have all the evidence I need to may an informed decision. Thing is, I figured this out for myself several years back before I read a single thread about how bad DxO scoring is. I did it by reading how they do the testing and how they arrive at there scores. DxO provided this information to determine the flaw to their madness. It was easy with just a little intelligent reasoning to realize the overall scores are flawed. Did I need to consult someone other than DxO to make this determination?

Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?

 

[elflord]

Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.

Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?

I don't own a Nikon. I own a 5D Mk II and a Panasonic GF2. Before that I owned a Rebel. I am not a "fan" of either Canon, Panasonic or any other brand.

I defend them because in my opinion they do an excellent job at what they do, and most of the attacks are not well founded (the vast majority of criticism comes from camera "fans" who are unhappy that their manufacturer doesn't do well on the test scores. That and I don't have a very high opinion of camera "fans".

 

[PackLight]

"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.

It has nothing to do with the Nikon/Canon debate as far as I am concerned.

You can read at DxO which numbers and factors they use. There are factors they leave out of the evaluation as well and they tell you this. The weight they apply to the three factors they use to arrive at the number score, is arrived at and determined by DxO. Why should we believe that the way they arrive at this average score is correct for how a sensor should be evaulated and scored? Is it because they "claim" to be the leader in sensor testing?

When I first went to DxO to use the information they provide to compare Canon cameras it was years ago and, Nikon only had few in the top ten. At that time when I looked at the scores they had just on Canon cameras the scores just didn't line up with reality with Canon vs Canon. If the final results do not line up with the real world situation, perhaps they should look at how they combine all this data to arrive at a final number score.

Then there is the description "landscape" or "sports" they use, rather than calling the test what it is actually for. Why don't they describe the test for what it is, and then imply this is important for that activity. With landscaping DR is not the only thing that matters, why apply the name landscape to high DR?

I find DxO's presentation of the data to miss leading. Any time a large company chooses to put out data that on the surface appears to miss lead the general public I think you have to question their motives.

I do not believe the "simply summarizing" view. But then maybe in their minds the believe they are.

 

[tnargs]

I do not believe the "simply summarizing" view. But then maybe in their minds they believe they are.

Have you seen their new lens scoring thingy? :

Clearly they haven't learnt their lesson. Blinkers? 8)

 

[PackLight]

I will se that day Canon produces a camera with as good or better sensor measurements than the Sony etc , how will you detractors of DXO look at DXO measurements then??

The exact same way I have for years. Even before Nikon released all of the recent cameras that scored so high and Canon was on the top. Their method was wrong in the past, it is now and unless they change it will be wrong in the future no matter who's sensor scores a better make believe score.

Will you switch your view of DxO when Canon is on top of the chart?

 

[elflord]

You can read at DxO which numbers and factors they use. There are factors they leave out of the evaluation as well and they tell you this. The weight they apply to the three factors they use to arrive at the number score, is arrived at and determined by DxO. Why should we believe that the way they arrive at this average score is correct for how a sensor should be evaulated and scored? Is it because they "claim" to be the leader in sensor testing?

The way they choose to weight those factors is subjective. Therefore it is neither correct, nor incorrect. If you demand hard numbers as opposed to the wishy-washy subjective stuff, you will insist on seeing the measurements. I would feel very differently about DxO if they only showed us their summary numbers and not the underlying measurements.

As far as I'm concerned, they are welcome to publish whatever subjective summaries they like, though I don't care for these myself. Ultimately these I see as just another opinion -- well informed of course, but still just an opinion. There are other smart people who will have different opinions. The only opinion I care much for is my own after seeing the data.

When I first went to DxO to use the information they provide to compare Canon cameras it was years ago and, Nikon only had few in the top ten.

So I don't understand how they are trying to be "misleading" or what their alleged purpose is.

Then there is the description "landscape" or "sports" they use, rather than calling the test what it is actually for. Why don't they describe the test for what it is, and then imply this is important for that activity. With landscaping DR is not the only thing that matters, why apply the name landscape to high DR?

I think the full name of the tests are (the parenthetical remarks are part of the name)

portrait (color depth),
landscape (dynamic range),
ports (low-light ISO)

I don't see what the problem is -- those who know what the technical terms mean, you can simply look past the non-technical ones. For those who don't, omitting the non-technical terms doesn't help a whole lot.

I find DxO's presentation of the data to miss leading. Any time a large company chooses to put out data that on the surface appears to miss lead the general public I think you have to question their motives.

I still don't get who they are misleading and what they are trying to get their "victims" to do. So far we don't have a plausible motive for this alleged deception. According to the originator of this thread, no-one is fooled anyway, so not only we don't have a cause, we don't see an effect either. All we have is some grumbling from disgruntled camera fans, and additionally the fact that you take issue with some perceived inaccuracies in the way they summarize their data.

 

[elflord]

Will you switch your view of DxO when Canon is on top of the chart?

No, but I might upgrade my 5DII.

 

[TheSuede]

Have you seen their new lens scoring thingy? :

Clearly they haven't learnt their lesson. Blinkers? 8)

I very much do NOT like their new lens score "thingy", and I especially dislike the fact that they removed the underlying base data from public view. But I know how to read MTF / image height and MTF / frequency diagrams - something I (without meaning to sound condescending) honestly believe very few average camera buyers do. I understand why they did it, but I don't like it.

 

[TheSuede]

Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

(some stuff of the later part of the quote removed)

Actually (actually x3 now! ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.

 

[PackLight]

I will se that day Canon produces a camera with as good or better sensor measurements than the Sony etc , how will you detractors of DXO look at DXO measurements then??

The exact same way I have for years. Even before Nikon released all of the recent cameras that scored so high and Canon was on the top. Their method was wrong in the past, it is now and unless they change it will be wrong in the future no matter who's sensor scores a better make believe score.

Will you switch your view of DxO when Canon is on top of the chart?

your answers and statements are not worthy to be treated with a answer
there are plenty answers already.

Why not? I answered your worthless question with a legitimate response.

 

[Hobby Shooter]

Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

(some stuff of the later part of the quote removed)

Actually (actually x3 now! ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.

I would argue that irrelevant can not be graded in %, it's either relevant or irrelevant, binary. On the other hand something can be more or less relevant, but something can not be more or less irrelevant.

 

[neuroanatomist]

Actually (actually x3 now! ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.

Perhaps one more refinement: irrelevant to camera sensor image quality. A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

 

[PackLight]

I still don't get who they are misleading and what they are trying to get their "victims" to do. So far we don't have a plausible motive for this alleged deception. According to the originator of this thread, no-one is fooled anyway, so not only we don't have a cause, we don't see an effect either. All we have is some grumbling from disgruntled camera fans, and additionally the fact that you take issue with some perceived inaccuracies in the way they summarize their data.

Not everyone researches as in depth as those on Canon Rumors. I know many with DSLR's that wouldn't have a clue how to interpret DxO's information, and they would just take the overall number at face value. What DxO's motive is I wouldn't know, but when ever a public company offers information there is usually a motive and it almost always involves monetary gain. Why do they post this information at all? Really I don't care.