December 05, 2016, 03:55:38 PM

Author Topic: DXO uh-oh?  (Read 70487 times)

unfocused

  • Canon EF 400mm f/2.8L IS II
  • *********
  • Posts: 3302
    • Mark Gordon Communications
Re: DXO uh-oh?
« Reply #60 on: July 27, 2014, 11:10:43 AM »
Why does everyone respond to dilbert's nonsense? Can't we just ignore his posts and hope he goes away? It would make this forum much more enjoyable.

It is bit extreme to stop anyone from posting their viewpoints.

The fun part is trying to get jrista/neuro to be open with people rather than hide their viewpoints and thoughts. "DxO and Nikon are joined at the hip". How many times has that been repeated now but no substance has been given as to why anyone should think that but yet nobody wants to back away from saying that.

An alternative explanation is that most people really don't care one way or the other.

The inconvenient truth is that DSLR technology has progressed to the point where differences between brands and even between formats is so insignificant that it seldom, if ever, has any real world impact on the final product – the photograph.

This forum provides daily proof of Sayre's law: "the intensity of feeling is inversely proportional to the value of the issues at stake."

Perhaps DxO is biased. Perhaps Nikon and Sony have decided to "build to the test." Perhaps the differences being tested are so insignificant that the ratings have only academic and no real-world application. Most likely it's a combination of all three.

It's not like the scores have the tiniest bit of impact on the market. So really, who cares?

canon rumors FORUM

Re: DXO uh-oh?
« Reply #60 on: July 27, 2014, 11:10:43 AM »

Sporgon

  • Canon EF 400mm f/2.8L IS II
  • *********
  • Posts: 3147
  • 5% of gear used 95% of the time
    • www.buildingpanoramics.com
Re: DXO uh-oh?
« Reply #61 on: July 27, 2014, 11:43:03 AM »
It's not like the scores have the tiniest bit of impact on the market. So really, who cares?

Or even an impact on practical use. If the there was any real, practical value to their 'metrics' photographers serious about low ISO performance would be deserting to Sony and Nikon in their droves, yet they are not because in the vast majority of low ISO circumstances there is just no difference, despite all the crap about read noise levels, FPN etc.

I remember he-who-shall-not-be-named once posted two identical shots from a 5DII and a D800 to show, in his opinion how much better the shadows were from the Nikon, but in his blinkered vision of pulled shadows he had overlooked the noise in the blue sky from the Nikon ! When I pointed this out there was a very hasty edit  ;D

So who cares ? Well unfortunately there are a growing number of web based review sites that quote DxO, perhaps because of the way in which DxO present their data; it's seen as being very scientific. Obviously to date this has had no detrimental impact on Canon's sales, so it would seem that at the moment the majority of purchasers don't take any notice of what they are saying, but I wonder if in time it could start to impact, but I suppose by then Canon may have a sensor that scores better on DxO.

MacroBug

  • PowerShot G1 X II
  • ***
  • Posts: 29
Re: DXO uh-oh?
« Reply #62 on: July 27, 2014, 11:58:53 AM »
sanj, I never stated dilbert couldn't post. However, we control how and when we respond to his posts. I noticed he has not addressed any of the unemotional, logical points made in a number of posts regarding lens ratings. He continually beats the 'joined at the hip' statement to death which got old in the first 2 pages of this thread. I learn a tremendous amount through this forum but not in this thread. I'm out on the rest of this one...

MLfan3

  • Rebel T6i
  • ****
  • Posts: 117
Re: DXO uh-oh?
« Reply #63 on: July 27, 2014, 02:58:43 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

jrista

  • Canon EF 600mm f/4L IS II
  • **********
  • Posts: 5334
  • EOL
    • Nature Photography
Re: DXO uh-oh?
« Reply #64 on: July 27, 2014, 04:07:30 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

Don Haines

  • Canon EF 600mm f/4L IS II
  • **********
  • Posts: 5437
  • posting cat pictures on the internet since 1986
Re: DXO uh-oh?
« Reply #65 on: July 27, 2014, 04:11:00 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?
I would be a lot happier with them if they left out the magic ratings numbers.....
The best camera is the one in your hands

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 19997
Re: DXO uh-oh?
« Reply #66 on: July 27, 2014, 04:18:11 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

Perhaps they should submit their 'science' to the Journal of Irreproducible Results.  They may even be worthy of consideration for an IgNobel Prize.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: DXO uh-oh?
« Reply #66 on: July 27, 2014, 04:18:11 PM »

Orangutan

  • 1D X Mark II
  • *******
  • Posts: 1557
Re: DXO uh-oh?
« Reply #67 on: July 27, 2014, 05:18:54 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

I agree.  Science demands transparency so the measurements can be replicated by others.  Little about their process is transparent.  It's not scientific, it's almost pseudo-scientific.

horshack

  • Canon AE-1
  • ***
  • Posts: 69
Re: DXO uh-oh?
« Reply #68 on: July 27, 2014, 05:41:12 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

Perhaps they should submit their 'science' to the Journal of Irreproducible Results.  They may even be worthy of consideration for an IgNobel Prize.
DxO documents their sensor testing procedure here:
http://www.dxomark.com/About/In-depth-measurements/DxOMark-testing-protocols/Noise-dynamic-range

DxO results have been independently reproduced at various times. For example:
http://www.dpreview.com/forums/post/33806693
http://www.dpreview.com/forums/post/33833501

MLfan3

  • Rebel T6i
  • ****
  • Posts: 117
Re: DXO uh-oh?
« Reply #69 on: July 27, 2014, 06:33:27 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

so what are the so-called incorrect results?
I think all what they have posted are right, at least mirror to my own experience.  only one issue I found with DXO mark is their stupid overall score D810=97, D800E =96, 5D3=81,etc.   But other than that almost all graphs and numbers they provided there seem very correct. Why do you think they are inconsistent with some obviously incorrect results?
but I think you know much more than me in this kind of things , so I would like to hear your view on DXO.

 

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 19997
Re: DXO uh-oh?
« Reply #70 on: July 27, 2014, 06:37:08 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

Perhaps they should submit their 'science' to the Journal of Irreproducible Results.  They may even be worthy of consideration for an IgNobel Prize.
DxO documents their sensor testing procedure here:
http://www.dxomark.com/About/In-depth-measurements/DxOMark-testing-protocols/Noise-dynamic-range

DxO results have been independently reproduced at various times. For example:
http://www.dpreview.com/forums/post/33806693
http://www.dpreview.com/forums/post/33833501

Your DxO link describes one of their Measurements, which as I've stated on multiple occasions (at least dozens, if not hundreds on these forums) I find generally well done and useful (except when they make errors and deny it, which seems to occur mainly in their lens tests).  The problems are not with their Measurements, but with their Scores.  Can you provide a link where DxO explicitly describes how their Scores are calculated from the Measurements?  No, because they don't disclose the specifics of how those Scores are calculated.  Nor do they explicitly describe the bias inherent in their Scores.

FWIW, Peter van den Hamer suggests an approximation he states usually falls within 1-2 points: DxOMark_Sensor_Score = 59 + 4.3*(ColorDepth-21.1) + 3.4*(DynamicRange-11.3) + 4.4*log2(ISO/663) -0.2.  He also states, "My guess is that the actual formula is non-linear and may use (under some conditions) coefficients of 5/5/5 rather than 4.3/3.4/4.4."  His suggestion that the 'master formula' which DxO uses may be modified under some conditions further supports the claim that DxO's scoring is biased.  Yeah, that sounds like good science. NOT.

As for your 'independent reproduction,' I clicked your first link but to be honest, I stopped reading after, "For some obscure reason - sunspots or moon phase or other strangeness - photons are behaving better today, and I achieved higher FWC results for my D3's than I have before."  Sorry, but independent verification of poor pseudoscience with worse pseudoscience is even less valid than two wrongs making a right.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

horshack

  • Canon AE-1
  • ***
  • Posts: 69
Re: DXO uh-oh?
« Reply #71 on: July 27, 2014, 07:37:47 PM »
as a multi-system user I have to agree with DXO guys, they are honest, much more so than DPR or any other unscientific review sites online.

The problem is that DXO's "science" is in dispute. How can you trust something that produces inconsistent and obviously incorrect results?

Perhaps they should submit their 'science' to the Journal of Irreproducible Results.  They may even be worthy of consideration for an IgNobel Prize.
DxO documents their sensor testing procedure here:
http://www.dxomark.com/About/In-depth-measurements/DxOMark-testing-protocols/Noise-dynamic-range

DxO results have been independently reproduced at various times. For example:
http://www.dpreview.com/forums/post/33806693
http://www.dpreview.com/forums/post/33833501

Your DxO link describes one of their Measurements, which as I've stated on multiple occasions (at least dozens, if not hundreds on these forums) I find generally well done and useful (except when they make errors and deny it, which seems to occur mainly in their lens tests).  The problems are not with their Measurements, but with their Scores.  Can you provide a link where DxO explicitly describes how their Scores are calculated from the Measurements?  No, because they don't disclose the specifics of how those Scores are calculated.  Nor do they explicitly describe the bias inherent in their Scores.
You can read about the methodology of their scores here:
http://www.dxomark.com/About/Sensor-scores

And I describe in detail their low-light score, the score which typically produces the most Canon vs Nikon controversy in online debates:
http://www.dpreview.com/forums/post/41265241

As for your 'independent reproduction,' I clicked your first link but to be honest, I stopped reading after, "For some obscure reason - sunspots or moon phase or other strangeness - photons are behaving better today, and I achieved higher FWC results for my D3's than I have before."  Sorry, but independent verification of poor pseudoscience with worse pseudoscience is even less valid than two wrongs making a right.
She wrote that as tongue 'n cheek, and it actually represents a sign of humility and willingness to be open to contrary points of view, signs of a good engineer/scientist. As for her credentials, if you follow her posts on dpreview you'll see she one of the most informed technical minds for camera sensor info. To cite a specific example, she reverse-engineered Nikon's long-exposure noise algorithm, identified serious problems with it, devised a much improved alternate algorithm which was relayed to Nikon by Thom Hogan and then later adopted by Nikon in subsequent camera designs.

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 19997
Re: DXO uh-oh?
« Reply #72 on: July 27, 2014, 07:45:54 PM »
You can read about the methodology of their scores here:
http://www.dxomark.com/About/Sensor-scores
One of the tenets of scientific research is that you publish your methods in sufficient detail that someone knowledgable in the field can repeat your experiments and derive equivalent results.  Sorry, on the page you linked or elsewhere on their site, I cannot find where they state the formula used to calculate their scores.  If that is an oversight on my part, can you please link to where they publish that part of their methods?  If not, my claim of their "Image Science" being poor pseudoscience remains valid.

She wrote that as tongue 'n cheek, and it actually represents a sign of humility and willingness to be open to contrary points of view, signs of a good engineer/scientist.
So your contention is that to paraphrase her post, 'I cannot get consistent absolute measurements from one day to the next, so I'll present relative data instead,' is the sign of a good scientist/engineer?  Sorry, but I develop and validate assays for a living, and absolute data with significant inter-run or inter-day variability means a poor assay that needs to be corrected appropriately, if possible (and if not, an alternate assay must be developed).

What I am getting from your posts is a better idea of why your 'dot tune' method doesn't stand the test of independent validation, at least in my hands.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: DXO uh-oh?
« Reply #72 on: July 27, 2014, 07:45:54 PM »

Orangutan

  • 1D X Mark II
  • *******
  • Posts: 1557
Re: DXO uh-oh?
« Reply #73 on: July 27, 2014, 07:46:08 PM »
You can read about the methodology of their scores here:
http://www.dxomark.com/About/Sensor-scores

I'm sorry, could you point out where the formula is given?  I see a few hints about what's considered important, but not the actual formula for the score.  All I was able to find was this:

Quote
How is Sensor Overall Score measured?The Sensor Overall Score is an average of the Portrait Score based on color depth, the Landscape Score based on dynamic range, and the Sports Score based on low-light ISO.



It says "an average:" presumably that's a weighted average, but they don't give the weightings.

Thanks.

bdunbar79

  • Canon EF 400mm f/2.8L IS II
  • *********
  • Posts: 3111
Re: DXO uh-oh?
« Reply #74 on: July 27, 2014, 07:48:03 PM »
<She wrote that as tongue 'n cheek, and it actually represents a sign of humility and willingness to be open to contrary points of view, signs of a good engineer/scientist. As for her credentials, if you follow her posts on dpreview you'll see she one of the most informed technical minds for camera sensor info. To cite a specific example, she reverse-engineered Nikon's long-exposure noise algorithm, identified serious problems with it, devised a much improved alternate algorithm which was relayed to Nikon by Thom Hogan and then later adopted by Nikon in subsequent camera designs.>

Right.  Scientific GARBAGE.  There are many scientists on this forum, myself included.  This doesn't count, sorry.  In science you don't get to "tongue 'n cheek" or get it right the majority of the time.  Either you do good science that's meaningful or you don't.  DxO mark does NOT.  We've all read that link and they do NOT disclose how scores are done/derived from the measurements. 

Besides, DxO mark isn't relevant.  Despite them scoring Nikon/Sony higher and higher against Canon product head to head, Canon still went from a 4% market share lead 4 years ago to a now 20% market share lead.  Nobody cares or nobody believes because of just that:  The garbage "science" they are doing.
2 x 1DX
B1G, MAC, GLIAC

canon rumors FORUM

Re: DXO uh-oh?
« Reply #74 on: July 27, 2014, 07:48:03 PM »