October 01, 2014, 07:01:19 PM

Author Topic: DxOMark vs. Reality  (Read 79733 times)

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #180 on: December 19, 2012, 07:03:33 PM »
How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #180 on: December 19, 2012, 07:03:33 PM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #181 on: December 19, 2012, 07:15:33 PM »

your answer is  clever like  as -  all Canons gear is worthless
Please point why and  how the DXO sensor measurement is faulty.

Which is a better overall camera?  The D4 or the D600?

Which will take the best picture?  The D4 or the D600?

DxO do not attempt to measure which is "a better overall camera" (whatever that means though I think most would agree that it's the D4) nor do they measure "which will take the best picture" (what does "the best picture" mean ?)

They benchmark sensors.

Now, your answer to the question is ?

verysimplejason

  • 1D X
  • *******
  • Posts: 1352
    • View Profile
    • My Flickr Account
Re: DxOMark vs. Reality
« Reply #182 on: December 19, 2012, 08:04:42 PM »
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14536
    • View Profile
Re: DxOMark vs. Reality
« Reply #183 on: December 19, 2012, 08:21:29 PM »
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores

Yep, reality is the whole point.  If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs.  Has that been happening?  Or does Canon still have a greater market share?
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

verysimplejason

  • 1D X
  • *******
  • Posts: 1352
    • View Profile
    • My Flickr Account
Re: DxOMark vs. Reality
« Reply #184 on: December 19, 2012, 08:32:25 PM »
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance:  nada more.

DXO scores sensors while in REALITY there's more to a camera than a sensor.  Is it that hard to understand?  That's the problem with DXO and what's almost everybody is pointing out.  DXO seems content on grading the sensor but not the whole camera body.  That's why this is titled DXOMARK vs REALITY.

TheSuede

  • PowerShot G1 X II
  • ***
  • Posts: 54
    • View Profile
Re: DxOMark vs. Reality
« Reply #185 on: December 19, 2012, 08:44:35 PM »
The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

As stated so many times before, you cannot put a "score" on a camera as a complete system for any sort of user average, you have to specify a quite tight definition of the score scope - what it does include and what it doesn't. And it does get tricky when you try to include more than one parameter in a "score".

You may (in my opinion...) for instance put a reasonably real-world relevant score on "low light noise performance". This score will then preferably accurately represent how much noise a camera will show in a shot given an average exposure of exactly "X" lumen seconds per mm2. The measurement "noise per lm*s/mm2" is just a strict definition of what a photographer would call "I want 1/200s at F2.8 in this light".

-But it (the score) will not say anything at all about how well the camera can AF in that same situation. Neither will it say anything about the viewfinder, the fps rate or the color accuracy. The score will be "low light NOISE performance", not "low light performance".
...

But the individual measurement scales and results in themselves that DxO use to build their total "score" are almost beyond reproach. You could possibly fault their presentation mode in some of the results, but not the results in themselves.
The few individuals and organizations that can replicate the accuracy and scientific stringency of their tests all arrive at the same conclusion - there's not much to be said about it except for the fact that DxO are almost 100% repeatable and accurate. The numbers just fit, and anyone measuring the same thing on the same cameras will arrive at the same result. And anyone comparing two cameras out in the real world can - if they adhere to reality in stead of maker-defined bogus ISO values and so on - also visualize exactly what that number means in an image.

So it's up to the reader of the results to make use of the information in a way that he or she can relate to, and here's where most people fall flat on their faces. Unfortunately DxO trip themselves up by trying to apply an overall "score" on a camera, but as far as anyone in the publishing business can tell - you NEED a promise of an end score or other simple graphical or single-digit grading system to get 95% of the readers to even bother starting to read.

If the average readers realizes that they actually might have to think a bit for themselves to get something out of something, they quite often just bugger off somewhere else before even starting to assimilate any real information.

TheSuede

  • PowerShot G1 X II
  • ***
  • Posts: 54
    • View Profile
Re: DxOMark vs. Reality
« Reply #186 on: December 19, 2012, 08:53:38 PM »
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

kind of like saying
:"Everything Apple makes right now has to be crap, they're constantly dropping in market share"
???

Neither of those arguments are worth paying any attention to, since there's more to sales and market shares than measurable product quality.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #186 on: December 19, 2012, 08:53:38 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14536
    • View Profile
Re: DxOMark vs. Reality
« Reply #187 on: December 19, 2012, 09:12:48 PM »
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

I could not agree more.  This was what I was driving at, initially.  At the time I originated this thread, there was a plethora of posts/threads proclaiming the superiority of the D800 (and other Nikon bodies) as cameras because of the DxOMark scores, and nearly as many posts/threads suggesting that Canon was doomed as a corporate entity because of their subpar sensors.  The former claim is clearly unsupported by DxOMark, and the latter claim is equally untrue.

Thanks, TheSuede, for your cogent post.   :)
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

thepancakeman

  • 6D
  • *****
  • Posts: 457
  • If at first you don't succeed, don't try skydiving
    • View Profile
Re: DxOMark vs. Reality
« Reply #188 on: December 19, 2012, 10:10:04 PM »
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance:  nada more.

The reality is that low-light ISO does not = a good sports camera, but that is how it's being portrayed.

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #189 on: December 19, 2012, 10:14:53 PM »
Yep, reality is the whole point.  If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs.  Has that been happening?  Or does Canon still have a greater market share?

It is not as the thread title suggests, a "competitor" or an "alternative" to reality but rather a subset thereof.

BTW, if the scores do not substantially affect sales, why all the concern about them "misleading" the "consumer" ?

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #190 on: December 19, 2012, 11:41:15 PM »
How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.

Fudging? It wasn't my word.

I have all the evidence I need to may an informed decision. Thing is, I figured this out for myself several years back before I read a single thread about how bad DxO scoring is. I did it by reading how they do the testing and how they arrive at there scores. DxO provided this information to determine the flaw to their madness. It was easy with just a little intelligent reasoning to realize the overall scores are flawed. Did I need to consult someone other than DxO to make this determination?

Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?
« Last Edit: December 19, 2012, 11:45:50 PM by PackLight »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #191 on: December 20, 2012, 06:09:19 AM »
Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.

Quote
Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?

I don't own a Nikon. I own a 5D Mk II and a Panasonic GF2. Before that I owned a Rebel. I am not a "fan" of either Canon, Panasonic or any other brand.

I defend them because in my opinion they do an excellent job at what they do, and most of the attacks are not well founded (the vast majority of criticism comes from camera "fans" who are unhappy that their manufacturer doesn't do well on the test scores. That and I don't have a very high opinion of camera "fans".

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #192 on: December 20, 2012, 09:16:02 AM »
"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.


It has nothing to do with the Nikon/Canon debate as far as I am concerned.

You can read at DxO which numbers and factors they use. There are factors they leave out of the evaluation as well and they tell you this. The weight they apply to the three factors they use to arrive at the number score, is arrived at and determined by DxO. Why should we believe that the way they arrive at this average score is correct for how a sensor should be evaulated and scored? Is it because they "claim" to be the leader in sensor testing?

When I first went to DxO to use the information they provide to compare Canon cameras it was years ago and, Nikon only had few in the top ten. At that time when I looked at the scores they had just on Canon cameras the scores just didn't line up with reality with Canon vs Canon. If the final results do not line up with the real world situation, perhaps they should look at how they combine all this data to arrive at a final number score.

Then there is the description "landscape" or "sports" they use, rather than calling the test what it is actually for. Why don't they describe the test for what it is, and then imply  this is important for that activity. With landscaping DR is not the only thing that matters, why apply the name landscape to high DR?

I find DxO's presentation of the data to miss leading. Any time a large company chooses to put out data that on the surface appears to miss lead the general public I think you have to question their motives.

I do not believe the "simply summarizing" view. But then maybe in their minds the believe they are.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #192 on: December 20, 2012, 09:16:02 AM »

tnargs

  • Rebel T5i
  • ****
  • Posts: 138
    • View Profile
Re: DxOMark vs. Reality
« Reply #193 on: December 20, 2012, 06:39:38 PM »
I do not believe the "simply summarizing" view. But then maybe in their minds they believe they are.

Have you seen their new lens scoring thingy?   ::)

Clearly they haven't learnt their lesson. Blinkers?  8)

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #194 on: December 20, 2012, 06:58:33 PM »
I will se that day  Canon produces a camera with as good or better sensor  measurements than the Sony etc , how will you detractors of DXO look at DXO  measurements then??


The exact same way I have for years. Even before Nikon released all of the recent cameras that scored so high and Canon was on the top. Their method was wrong in the past, it is now and unless they change it will be wrong in the future no matter who's sensor scores a better make believe score.

Will you switch your view of DxO when Canon is on top of the chart?


canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #194 on: December 20, 2012, 06:58:33 PM »