December 07, 2016, 11:50:17 AM

Author Topic: Why the DxO bashing?  (Read 67727 times)

Mt Spokane Photography

  • Zeiss 50mm F 0.7
  • ***********
  • Posts: 12372
Re: Why the DxO bashing?
« Reply #30 on: February 07, 2014, 08:10:28 PM »

I've been on a rampage against DXO lens tests for almost as long as they've been around, and I've been QUITE vocal about that here in these forums. DXO lens tests frequently indicate that Canon lenses are better than the competitions, DESPITE the undue bias they give to Nikon lenses thanks to the D800. Doesn't change the fact that DXO's lens tests are a joke, again thanks to that "weighted scoring" they do that vastly overweights factors that don't play a big roll in IQ, and vastly underweights factors that do play a big role. They also use the word "transmission" to refer to what is really "aperture", and therefor ALL of their lens tests are massively skewed by the transmission factor.


Agreed!
 
I have a D300s and a Nikon 200-400mm VRI, which is a nice lens by anyone but DXO's measurements.  They tested the VRII model which is basically the same lens with improved VR and give it a score of 12! 
http://www.dxomark.com/Lenses/Nikon/Nikon-AF-S-Nikkor-200-400mm-F4G-ED-VR-II-mounted-on-Nikon-D300s__614
 
Now, if you put it on a D7000, it jumps to 14.  On a D3 its 17, and on a D700 its also 17.  On a D4, it jumps to 21, on a D600, it jumps to 24, and on a D800, its 25!
This is why you don't compare lenses across different camera models, much less across different brands.  Testing lenses on a camera body gives results that only apply to that body or one with the same sensor.  In general, the test methods will give higher numbers with more MP.  The lens itself did not change and is no better or worse just because its on a camera with more pixels.
The reason is simple, the MTF of a image is a product of the MTF of the lens, The Body, the monitor or printer its viewed on, and even your eyes.  Raising the MTF of any of these things will improve the image as long as the others don't change.  Of course, DXO does not USE MTF,  just because the entire photography world uses it, they invented their own number, MPIX.
 
Now, if DXO wanted to compare lenses between Canon and Nikon, they'd test them all on the same Canon body.  That would give you at lease some comparison, but it still would not be accurate, since manufacturers cameras recognize a lens model and may make subtle adjustments to exposure at the edges.
 
That's why most knowledgeable lens testers provide a warning note that says don't compare on different models or manufacturers, a lens test on a D300s is only good for a D300s, but may be similar for a body with the same sensor.
« Last Edit: February 08, 2014, 04:57:21 PM by Mt Spokane Photography »

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #30 on: February 07, 2014, 08:10:28 PM »

rpt

  • Canon EF 300mm f/2.8L IS II
  • ********
  • Posts: 2626
Re: Why the DxO bashing?
« Reply #31 on: February 07, 2014, 09:12:38 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Does he have an MBA, by chance?
Nope! An ordinary engineer like me ;)
What? :o ... he is just an ordinary engineer and does not even have the highly acclaimed Masters in Bull$h!tting Arts? ... how dare he! ;D
Ha! Ha! Ha!

Don, thanks for the beautifully crafted sentence. It settled matters for me.

jrista

  • Canon EF 600mm f/4L IS II
  • **********
  • Posts: 5334
  • EOL
    • Nature Photography
Re: Why the DxO bashing?
« Reply #32 on: February 07, 2014, 10:04:47 PM »
[

I've been on a rampage against DXO lens tests for almost as long as they've been around, and I've been QUITE vocal about that here in these forums. DXO lens tests frequently indicate that Canon lenses are better than the competitions, DESPITE the undue bias they give to Nikon lenses thanks to the D800. Doesn't change the fact that DXO's lens tests are a joke, again thanks to that "weighted scoring" they do that vastly overweights factors that don't play a big roll in IQ, and vastly underweights factors that do play a big role. They also use the word "transmission" to refer to what is really "aperture", and therefor ALL of their lens tests are massively skewed by the transmission factor.



 
I have a D300s and a Nikon 200-400mm VRI, which is a nice lens by anyone but DXO's measurements.  They tested the VRII model which is basically the same lens with improved VR and give it a score of 12! 
http://www.dxomark.com/Lenses/Nikon/Nikon-AF-S-Nikkor-200-400mm-F4G-ED-VR-II-mounted-on-Nikon-D300s__614
 
Now, if you put it on a D7000, it jumps to 14.  On a D3 its 17, and on a D700 its also 17.  On a D4, it jumps to 21, on a D600, it jumps to 24, and on a D800, its 25!
This is why you don't compare lenses across different camera models, much less across different brands.  Testing lenses on a camera body gives results that only apply to that body or one with the same sensor.  In general, the test methods will give higher numbers with more MP.  The lens itself did not change and is no better or worse just because its on a camera with more pixels.
The reason is simple, the MTF of a image is a product of the MTF of the lens, The Body, the monitor or printer its viewed on, and even your eyes.  Raising the MTF of any of these things will improve the image as long as the others don't change.  Of course, DXO does not USE MTF,  just because the entire photography world uses it, they invented their own number, MPIX.
 
Now, if DXO wanted to compare lenses between Canon and Nikon, they'd test them all on the same Canon body.  That would give you at lease some comparison, but it still would not be accurate, since manufacturers cameras recognize a lens model and may make subtle adjustments to exposure at the edges.
 
That's why most knowledgeable lens testers provide a warning note that says don't compare on different models or manufacturers, a lens test on a D300s is only good for a D300s, but may be similar for a body with the same sensor.

Aye. All this I know. Which is why I always say: Standard lens testing is useless! It has no real value, since output resolution is usually sensor bound. These tests don't really tell you anything about the lens, and they aren't comparable...so....what value do they hold? Personally, I just made the decision to ditch lens tests entirely and rely on model-generated MTFs to determine how good a lens is. You can actually garner a LOT of useful knowledge about a lens from a single MTF chart, not the least of which is corner performance.

Anyway....

Albi86

  • 1D Mark IV
  • ******
  • Posts: 824
Re: Why the DxO bashing?
« Reply #33 on: February 08, 2014, 07:48:57 AM »
This thread is a festival of misinformation.

First of all, every test is based on 3 stages: design, data collection, data interpretation.

DXO is quite transparent on the way they design the tests and collect the data. They are not transparent at all in the way they interpret the results and produce a verdict (score). Are they biased? Sure. Every test is. No test can possibly encompass every scenario and variable; the testers decide what to focus on - in this case ISO 100, 8 MP, blah bla. You may agree or disagree with their choices, but your disagreement doens't make the test "stupid", because your choice is not worth any more than theirs.

It must be noticed, for example, that they don't test lenses per se, but lens-camera combinations; this is why lens scores change camera-to-camera. The sharpness of the final output depends on camera AND lens and so this is why they do it. Again, you may disagree. However, every lens will perform differently on a D700 and a D800; testing a lens on a D700 and saying that it's great it's no indication of how it will fare on a 3x resolution body. In this sense their measurements are far more accurate.

It should also be noted that their tests are based on a resized 8 MP file. This is why Canon's sharpness scores are often higher, and Nikon's scores are higher in most other fields. This is not the bread & butter of pixel peepers, but it can better reflect a real world scenario of printed pictures. Again, you may disagree, but it stands true that you need around 8 MP for a 300dpi A4 print, so how the final output will look like can be a more important information than a 100% crop.

Ranting about something while swearing that it's useless and meaningless, is paradoxical and childish. For as much as I agree with most people saying that the way DXO interprets results is disputable and pretty much useless, the data they collect is quite good. You can have DR graphs at different ISOs, compare screen and print output from different cameras and lenses, etc. All of these data are freely accessible and everyone can then draw his own conclusions - probably a more interesting endeavour than just ranting about DXO's.

Every test is useful in its limited purpose. Ranting derives from 2 major causes:
- Incapability of understanding the test methodology, and thus both its usefulness and limitations: this leads to labeling as stupid.
- Results are not what one wants to hear: this leads to discrediting the tests, claiming bias and second interests, etc etc.

I agree with those who thinks that, if it was the other way around, this forum would be full of DXO ambassadors. Same as the recent "conversion" of Scott Kelby to Canon has not raised any suspicions about the actual circumstances as they were presented; it has been a genuine event of a prodigal son finally seeing the true light. 

Keith_Reeder

  • Canon 70D
  • ****
  • Posts: 302
Re: Why the DxO bashing?
« Reply #34 on: February 08, 2014, 08:27:53 AM »
As a result, all sorts of reasons have been created by the Canon fanbase as to why this is and how DxO is useless but rest assured, if the shoe were on the other foot, people would be lauding DxO.

You know, I've quit this forum once, entirely because of this interminable, revisionist whining from you and LetTheRightTrollIn, and now I find myself having to sign up again just to address this drivel.

What's the real problem with DxO?

People like you.

The fact is that nobody disputes that some Nikons and Sonys have a low ISO DR advantage. Nobody.

But that's not good for you and you OCD mission to be right at all costs: you and your like have to bang away in a preposterous and utterly idiotic attempt to convince the world that this is all that matters.

Well get this through your head once and for all: for the vast majority of Canon users (and - truth be known - most Nikon, Sony and Pentax users too), it doesn't matter worth a damn.

But you simply can't accept that, can you? So instead simply letting DxO's findings stand for themselves, you have to make the DR difference into a crusade - for God knows what motivation, but it isn't healthy.

Again: we don't care about DxO's findings. They just don't matter.

Clear enough yet?

And if it did eventually transpire that Canon started getting the upper hand in the DxO DR race, Canon users wouldn't be "lauding" DxO because it proved that they had better cameras now, but purely to shut trolls like you up.

Your tedious obsession with the trivial difference between Canon's and other sensors hasn't stopped us from taking thousands, millions of incredible images; and besides, I've been able to open up shadows by multiple stops cleanly and easily from my 7D simply because I know how to choose and use my Raw converters and pixel editors properly.

As Americans are wont to say, "sucks to be you" -  but if you're seriously telling us that you can't take great pictures with your Canon cameras, or more to the point that the number that are spoiled for want of better low ISO shadow DR is too high to accept, I'm going to call you a bare-faced liar.

So, once more, for the avoidance of any doubt: the problem with DxO is obsessive, disingenuous, revisionist trolls like you. We're sick of having it shoved down our throats what should to be important to us, just because it's (supposedly) crucial to you.

(Doubtless this will be deleted, but while it stands, it makes the point).
« Last Edit: February 08, 2014, 08:37:58 AM by Keith_Reeder »

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 20026
Re: Why the DxO bashing?
« Reply #35 on: February 08, 2014, 09:06:15 AM »
This thread is a festival of misinformation.

First of all, every test is based on 3 stages: design, data collection, data interpretation.

DXO is quite transparent on the way they design the tests and collect the data. They are not transparent at all in the way they interpret the results and produce a verdict (score). Are they biased? Sure. Every test is. No test can possibly encompass every scenario and variable; the testers decide what to focus on - in this case ISO 100, 8 MP, blah bla. You may agree or disagree with their choices, but your disagreement doens't make the test "stupid", because your choice is not worth any more than theirs.

It must be noticed, for example, that they don't test lenses per se, but lens-camera combinations; this is why lens scores change camera-to-camera. The sharpness of the final output depends on camera AND lens and so this is why they do it. Again, you may disagree. However, every lens will perform differently on a D700 and a D800; testing a lens on a D700 and saying that it's great it's no indication of how it will fare on a 3x resolution body. In this sense their measurements are far more accurate.

It should also be noted that their tests are based on a resized 8 MP file. This is why Canon's sharpness scores are often higher, and Nikon's scores are higher in most other fields. This is not the bread & butter of pixel peepers, but it can better reflect a real world scenario of printed pictures. Again, you may disagree, but it stands true that you need around 8 MP for a 300dpi A4 print, so how the final output will look like can be a more important information than a 100% crop.

Ranting about something while swearing that it's useless and meaningless, is paradoxical and childish. For as much as I agree with most people saying that the way DXO interprets results is disputable and pretty much useless, the data they collect is quite good. You can have DR graphs at different ISOs, compare screen and print output from different cameras and lenses, etc. All of these data are freely accessible and everyone can then draw his own conclusions - probably a more interesting endeavour than just ranting about DXO's.

Every test is useful in its limited purpose. Ranting derives from 2 major causes:
- Incapability of understanding the test methodology, and thus both its usefulness and limitations: this leads to labeling as stupid.
- Results are not what one wants to hear: this leads to discrediting the tests, claiming bias and second interests, etc etc.

I agree with those who thinks that, if it was the other way around, this forum would be full of DXO ambassadors. Same as the recent "conversion" of Scott Kelby to Canon has not raised any suspicions about the actual circumstances as they were presented; it has been a genuine event of a prodigal son finally seeing the true light.

That post was a mini-festival of misinformation...

The key points are:

  • DxO does not disclose their methods for deriving their scores, which renders their scores meaningless
  • Some of DxO's measurements have errors, which makes all of their measurements suspect
  • DxO does not acknowledge their mistakes and issue corrigenda, but rather silently modify the original data, rendering their scientific ethics questionable

I'm not saying their information is useless, but I do suggest people view their measurements with caution, and ignore their Scores.

By the way, your recollection of facts is suspect, too.  "A genuine event of a prodigal son finally seeing the true light?"  Go back and re-read the Kelby conversion thread(s)...there was pretty broad support for the idea that he did it for the money.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Don Haines

  • Canon EF 600mm f/4L IS II
  • **********
  • Posts: 5447
  • posting cat pictures on the internet since 1986
Re: Why the DxO bashing?
« Reply #36 on: February 08, 2014, 09:14:49 AM »
By the way, your recollection of facts is suspect, too.  "A genuine event of a prodigal son finally seeing the true light?"  Go back and re-read the Kelby conversion thread(s)...there was pretty broad support for the idea that he did it for the money.
NO!!!!!!! Tell me it isn't so!!!!! Come on now, who ever heard of a business doing something for money! :)
The best camera is the one in your hands

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #36 on: February 08, 2014, 09:14:49 AM »

Albi86

  • 1D Mark IV
  • ******
  • Posts: 824
Re: Why the DxO bashing?
« Reply #37 on: February 08, 2014, 09:18:54 AM »

The key points are:

  • DxO does not disclose their methods for deriving their scores, which renders their scores meaningless
  • Some of DxO's measurements have errors, which makes all of their measurements suspect
  • DxO does not acknowledge their mistakes and issue corrigenda, but rather silently modify the original data, rendering their scientific ethics questionable


On which I totally agree. In fact I have specifically addressed the difference between data themselves and their interpretation, as well as pointing out the need of accepting the limitation of any single test and understanding the useful information it provides without labeling the whole thing as pointless, biased or false.

Orangutan

  • 1D X Mark II
  • *******
  • Posts: 1565
Re: Why the DxO bashing?
« Reply #38 on: February 08, 2014, 11:04:50 AM »
To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

Would you please post that raw file?

David Hull

  • EOS M3
  • ****
  • Posts: 234
Re: Why the DxO bashing?
« Reply #39 on: February 08, 2014, 11:32:45 AM »
If nothing else comes out of this discussion, at least we can hope that the folks at DxO will be reading this, too.

Nah.... it's not in French.

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 20026
Re: Why the DxO bashing?
« Reply #40 on: February 08, 2014, 12:11:31 PM »
To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

From your description, the scene would likely have had >15 stops of DR, supporting my earlier statement.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Mt Spokane Photography

  • Zeiss 50mm F 0.7
  • ***********
  • Posts: 12372
Re: Why the DxO bashing?
« Reply #41 on: February 08, 2014, 12:43:01 PM »

The key points are:

  • DxO does not disclose their methods for deriving their scores, which renders their scores meaningless
  • Some of DxO's measurements have errors, which makes all of their measurements suspect
  • DxO does not acknowledge their mistakes and issue corrigenda, but rather silently modify the original data, rendering their scientific ethics questionable

On which I totally agree. In fact I have specifically addressed the difference between data themselves and their interpretation, as well as pointing out the need of accepting the limitation of any single test and understanding the useful information it provides without labeling the whole thing as pointless, biased or false.

Mostly your post used the word rant many times and was a general insult to many of us who post here.  Just stick to your opinions about the tests and stop trying to insult members.  That's the stuff of Trolls.

tron

  • Canon EF 300mm f/2.8L IS II
  • ********
  • Posts: 2868

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #42 on: February 08, 2014, 01:30:06 PM »

David Hull

  • EOS M3
  • ****
  • Posts: 234
Re: Why the DxO bashing?
« Reply #43 on: February 08, 2014, 01:34:23 PM »
...
Many Canon Rumors trolls may experience great satisfaction from demonstrating that ~2 stop deficit in low ISO DR by shooting images with the lens cap on, then pushing those images 4-5 stops in post. 
...
Many times in the DR debate, those bashing Canon sensors have been asked to provide examples of shots ruined by Canon's 'poor low ISO DR' that would have been saved by those extra two stops.  Personally, I have almost no examples of that situation - in many scenes, the ~12 stops I get is sufficient, and when the scene DR is greater than 12 stops, it's almost always greater than 14 stops, too.

Dude, it's not just the extra DR but the rampant noise in the shadow details of Canon files. This all but makes the bottom stop or two useless.

To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

So to mimic what has been said about the dynamic range of the 5D Mark II (and by extension at least that of the 5D Mark III) "usable DR of the camera is one or two stops lower than what is measured due to the inability to use the shadows."
So what.  What I always find so interesting about these discussions is that at the end of the day, based on all the examples that people tend to put up (the photo you are referencing most likely no different) this superior "state of the art" sensor technology with all this extensive DR advantages has done very little to advance the state of "ART" with regard to photography.  If it really produced the dramatic advances in Image Quality that the proponents always claim, it would have gained significant traction in the market place.  If IQ were really a significant problem with Canon equipment as the Sony/Nikon proponents like to conclude, nobody would buy Canon product -- yet countless thousands of photographers have been able to use it with tremendous success despite this corner case limitation.

I think what really frustrates the Nikon/Sony fan-club is that despite what really is a significant difference in measurable performance between the system implementations chosen by the two manufacturers, it really is mostly a corner case issue and hasn't really proven to affect the bottom line enough to force Canon to address it. 

The endless barrage of poorly executed example images just hasn't gained the traction they expected.  The reaction to most of these (and there have been a boat load of them over the years) has been "yea... but why do I care".  The best example of this is probably the oft quoted Fred Miranda review where the reviewer shot two pages of magnificent images in Yosemite and could not produce a shot where the DR of the camera was a limitation -- to do that he gad to shoot something way less compelling.  Both sides seem to be reasonably satisfied with their choices.  Do I wish I had the same low shadow noise that I could have with a Sony sensor probably yes, do I wish Canon would solve it -- probably yes.  Has it ever gotten in my way, no not really.

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 20026
Re: Why the DxO bashing?
« Reply #44 on: February 08, 2014, 01:40:33 PM »
I think what really frustrates the Nikon/Sony fan-club is that despite what really is a significant difference in measurable performance between the system implementations chosen by the two manufacturers, it really is mostly a corner case issue and hasn't really proven to affect the bottom line enough to force Canon to address it. 

Pretty much what I said 1.5 years ago, and nothing has changed since then.

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.
EOS 1D X, EOS M2, lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #44 on: February 08, 2014, 01:40:33 PM »