July 28, 2014, 01:44:24 PM

Author Topic: Why the DxO bashing?  (Read 20767 times)

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3741
  • POTATO
    • View Profile
    • Nature Photography
Re: Why the DxO bashing?
« Reply #30 on: February 07, 2014, 02:59:09 PM »
People bash DxO because during the time when DxO results have been shared around more and more widely talked about on the 'Net, they always favour Nikon/Sony.

As a result, all sorts of reasons have been created by the Canon fanbase as to why this is and how DxO is useless but rest assured, if the shoe were on the other foot, people would be lauding DxO.

LOL. Your ignoring the facts here, bub. I've been on a rampage against DXO lens tests for almost as long as they've been around, and I've been QUITE vocal about that here in these forums. DXO lens tests frequently indicate that Canon lenses are better than the competitions, DESPITE the undue bias they give to Nikon lenses thanks to the D800. Doesn't change the fact that DXO's lens tests are a joke, again thanks to that "weighted scoring" they do that vastly overweights factors that don't play a big roll in IQ, and vastly underweights factors that do play a big role. They also use the word "transmission" to refer to what is really "aperture", and therefor ALL of their lens tests are massively skewed by the transmission factor.

It doesn't matter who DXO favors. Their science is too obscure, and the results are too skewed, to take any of their scores at face value without a significant dose of salt grains. It really has nothing to do with who they cater to, and everything to do with how they do things.
My Photography
Current Gear: Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: Canon 5D III/7D II | Canon EF 300mm f/2.8 L II

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #30 on: February 07, 2014, 02:59:09 PM »

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8261
    • View Profile
Re: Why the DxO bashing?
« Reply #31 on: February 07, 2014, 08:10:28 PM »

I've been on a rampage against DXO lens tests for almost as long as they've been around, and I've been QUITE vocal about that here in these forums. DXO lens tests frequently indicate that Canon lenses are better than the competitions, DESPITE the undue bias they give to Nikon lenses thanks to the D800. Doesn't change the fact that DXO's lens tests are a joke, again thanks to that "weighted scoring" they do that vastly overweights factors that don't play a big roll in IQ, and vastly underweights factors that do play a big role. They also use the word "transmission" to refer to what is really "aperture", and therefor ALL of their lens tests are massively skewed by the transmission factor.



Agreed!
 
I have a D300s and a Nikon 200-400mm VRI, which is a nice lens by anyone but DXO's measurements.  They tested the VRII model which is basically the same lens with improved VR and give it a score of 12! 
http://www.dxomark.com/Lenses/Nikon/Nikon-AF-S-Nikkor-200-400mm-F4G-ED-VR-II-mounted-on-Nikon-D300s__614
 
Now, if you put it on a D7000, it jumps to 14.  On a D3 its 17, and on a D700 its also 17.  On a D4, it jumps to 21, on a D600, it jumps to 24, and on a D800, its 25!
This is why you don't compare lenses across different camera models, much less across different brands.  Testing lenses on a camera body gives results that only apply to that body or one with the same sensor.  In general, the test methods will give higher numbers with more MP.  The lens itself did not change and is no better or worse just because its on a camera with more pixels.
The reason is simple, the MTF of a image is a product of the MTF of the lens, The Body, the monitor or printer its viewed on, and even your eyes.  Raising the MTF of any of these things will improve the image as long as the others don't change.  Of course, DXO does not USE MTF,  just because the entire photography world uses it, they invented their own number, MPIX.
 
Now, if DXO wanted to compare lenses between Canon and Nikon, they'd test them all on the same Canon body.  That would give you at lease some comparison, but it still would not be accurate, since manufacturers cameras recognize a lens model and may make subtle adjustments to exposure at the edges.
 
That's why most knowledgeable lens testers provide a warning note that says don't compare on different models or manufacturers, a lens test on a D300s is only good for a D300s, but may be similar for a body with the same sensor.
« Last Edit: February 08, 2014, 04:57:21 PM by Mt Spokane Photography »

rpt

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2119
  • Could not wait for 7D2 so I got the 5D3
    • View Profile
Re: Why the DxO bashing?
« Reply #32 on: February 07, 2014, 09:12:38 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Does he have an MBA, by chance?
Nope! An ordinary engineer like me ;)
What? :o ... he is just an ordinary engineer and does not even have the highly acclaimed Masters in Bull$h!tting Arts? ... how dare he! ;D
Ha! Ha! Ha!

Don, thanks for the beautifully crafted sentence. It settled matters for me.

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3741
  • POTATO
    • View Profile
    • Nature Photography
Re: Why the DxO bashing?
« Reply #33 on: February 07, 2014, 10:04:47 PM »
[


I've been on a rampage against DXO lens tests for almost as long as they've been around, and I've been QUITE vocal about that here in these forums. DXO lens tests frequently indicate that Canon lenses are better than the competitions, DESPITE the undue bias they give to Nikon lenses thanks to the D800. Doesn't change the fact that DXO's lens tests are a joke, again thanks to that "weighted scoring" they do that vastly overweights factors that don't play a big roll in IQ, and vastly underweights factors that do play a big role. They also use the word "transmission" to refer to what is really "aperture", and therefor ALL of their lens tests are massively skewed by the transmission factor.



 
I have a D300s and a Nikon 200-400mm VRI, which is a nice lens by anyone but DXO's measurements.  They tested the VRII model which is basically the same lens with improved VR and give it a score of 12! 
http://www.dxomark.com/Lenses/Nikon/Nikon-AF-S-Nikkor-200-400mm-F4G-ED-VR-II-mounted-on-Nikon-D300s__614
 
Now, if you put it on a D7000, it jumps to 14.  On a D3 its 17, and on a D700 its also 17.  On a D4, it jumps to 21, on a D600, it jumps to 24, and on a D800, its 25!
This is why you don't compare lenses across different camera models, much less across different brands.  Testing lenses on a camera body gives results that only apply to that body or one with the same sensor.  In general, the test methods will give higher numbers with more MP.  The lens itself did not change and is no better or worse just because its on a camera with more pixels.
The reason is simple, the MTF of a image is a product of the MTF of the lens, The Body, the monitor or printer its viewed on, and even your eyes.  Raising the MTF of any of these things will improve the image as long as the others don't change.  Of course, DXO does not USE MTF,  just because the entire photography world uses it, they invented their own number, MPIX.
 
Now, if DXO wanted to compare lenses between Canon and Nikon, they'd test them all on the same Canon body.  That would give you at lease some comparison, but it still would not be accurate, since manufacturers cameras recognize a lens model and may make subtle adjustments to exposure at the edges.
 
That's why most knowledgeable lens testers provide a warning note that says don't compare on different models or manufacturers, a lens test on a D300s is only good for a D300s, but may be similar for a body with the same sensor.


Aye. All this I know. Which is why I always say: Standard lens testing is useless! It has no real value, since output resolution is usually sensor bound. These tests don't really tell you anything about the lens, and they aren't comparable...so....what value do they hold? Personally, I just made the decision to ditch lens tests entirely and rely on model-generated MTFs to determine how good a lens is. You can actually garner a LOT of useful knowledge about a lens from a single MTF chart, not the least of which is corner performance.

Anyway....
My Photography
Current Gear: Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: Canon 5D III/7D II | Canon EF 300mm f/2.8 L II

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Why the DxO bashing?
« Reply #34 on: February 08, 2014, 07:48:57 AM »
This thread is a festival of misinformation.

First of all, every test is based on 3 stages: design, data collection, data interpretation.

DXO is quite transparent on the way they design the tests and collect the data. They are not transparent at all in the way they interpret the results and produce a verdict (score). Are they biased? Sure. Every test is. No test can possibly encompass every scenario and variable; the testers decide what to focus on - in this case ISO 100, 8 MP, blah bla. You may agree or disagree with their choices, but your disagreement doens't make the test "stupid", because your choice is not worth any more than theirs.

It must be noticed, for example, that they don't test lenses per se, but lens-camera combinations; this is why lens scores change camera-to-camera. The sharpness of the final output depends on camera AND lens and so this is why they do it. Again, you may disagree. However, every lens will perform differently on a D700 and a D800; testing a lens on a D700 and saying that it's great it's no indication of how it will fare on a 3x resolution body. In this sense their measurements are far more accurate.

It should also be noted that their tests are based on a resized 8 MP file. This is why Canon's sharpness scores are often higher, and Nikon's scores are higher in most other fields. This is not the bread & butter of pixel peepers, but it can better reflect a real world scenario of printed pictures. Again, you may disagree, but it stands true that you need around 8 MP for a 300dpi A4 print, so how the final output will look like can be a more important information than a 100% crop.

Ranting about something while swearing that it's useless and meaningless, is paradoxical and childish. For as much as I agree with most people saying that the way DXO interprets results is disputable and pretty much useless, the data they collect is quite good. You can have DR graphs at different ISOs, compare screen and print output from different cameras and lenses, etc. All of these data are freely accessible and everyone can then draw his own conclusions - probably a more interesting endeavour than just ranting about DXO's.

Every test is useful in its limited purpose. Ranting derives from 2 major causes:
- Incapability of understanding the test methodology, and thus both its usefulness and limitations: this leads to labeling as stupid.
- Results are not what one wants to hear: this leads to discrediting the tests, claiming bias and second interests, etc etc.

I agree with those who thinks that, if it was the other way around, this forum would be full of DXO ambassadors. Same as the recent "conversion" of Scott Kelby to Canon has not raised any suspicions about the actual circumstances as they were presented; it has been a genuine event of a prodigal son finally seeing the true light. 

Keith_Reeder

  • SX50 HS
  • **
  • Posts: 1
    • View Profile
Re: Why the DxO bashing?
« Reply #35 on: February 08, 2014, 08:27:53 AM »
As a result, all sorts of reasons have been created by the Canon fanbase as to why this is and how DxO is useless but rest assured, if the shoe were on the other foot, people would be lauding DxO.

You know, I've quit this forum once, entirely because of this interminable, revisionist whining from you and LetTheRightTrollIn, and now I find myself having to sign up again just to address this drivel.

What's the real problem with DxO?

People like you.

The fact is that nobody disputes that some Nikons and Sonys have a low ISO DR advantage. Nobody.

But that's not good for you and you OCD mission to be right at all costs: you and your like have to bang away in a preposterous and utterly idiotic attempt to convince the world that this is all that matters.

Well get this through your head once and for all: for the vast majority of Canon users (and - truth be known - most Nikon, Sony and Pentax users too), it doesn't matter worth a damn.

But you simply can't accept that, can you? So instead simply letting DxO's findings stand for themselves, you have to make the DR difference into a crusade - for God knows what motivation, but it isn't healthy.

Again: we don't care about DxO's findings. They just don't matter.

Clear enough yet?

And if it did eventually transpire that Canon started getting the upper hand in the DxO DR race, Canon users wouldn't be "lauding" DxO because it proved that they had better cameras now, but purely to shut trolls like you up.

Your tedious obsession with the trivial difference between Canon's and other sensors hasn't stopped us from taking thousands, millions of incredible images; and besides, I've been able to open up shadows by multiple stops cleanly and easily from my 7D simply because I know how to choose and use my Raw converters and pixel editors properly.

As Americans are wont to say, "sucks to be you" -  but if you're seriously telling us that you can't take great pictures with your Canon cameras, or more to the point that the number that are spoiled for want of better low ISO shadow DR is too high to accept, I'm going to call you a bare-faced liar.

So, once more, for the avoidance of any doubt: the problem with DxO is obsessive, disingenuous, revisionist trolls like you. We're sick of having it shoved down our throats what should to be important to us, just because it's (supposedly) crucial to you.

(Doubtless this will be deleted, but while it stands, it makes the point).
« Last Edit: February 08, 2014, 08:37:58 AM by Keith_Reeder »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13586
    • View Profile
Re: Why the DxO bashing?
« Reply #36 on: February 08, 2014, 09:06:15 AM »
This thread is a festival of misinformation.

First of all, every test is based on 3 stages: design, data collection, data interpretation.

DXO is quite transparent on the way they design the tests and collect the data. They are not transparent at all in the way they interpret the results and produce a verdict (score). Are they biased? Sure. Every test is. No test can possibly encompass every scenario and variable; the testers decide what to focus on - in this case ISO 100, 8 MP, blah bla. You may agree or disagree with their choices, but your disagreement doens't make the test "stupid", because your choice is not worth any more than theirs.

It must be noticed, for example, that they don't test lenses per se, but lens-camera combinations; this is why lens scores change camera-to-camera. The sharpness of the final output depends on camera AND lens and so this is why they do it. Again, you may disagree. However, every lens will perform differently on a D700 and a D800; testing a lens on a D700 and saying that it's great it's no indication of how it will fare on a 3x resolution body. In this sense their measurements are far more accurate.

It should also be noted that their tests are based on a resized 8 MP file. This is why Canon's sharpness scores are often higher, and Nikon's scores are higher in most other fields. This is not the bread & butter of pixel peepers, but it can better reflect a real world scenario of printed pictures. Again, you may disagree, but it stands true that you need around 8 MP for a 300dpi A4 print, so how the final output will look like can be a more important information than a 100% crop.

Ranting about something while swearing that it's useless and meaningless, is paradoxical and childish. For as much as I agree with most people saying that the way DXO interprets results is disputable and pretty much useless, the data they collect is quite good. You can have DR graphs at different ISOs, compare screen and print output from different cameras and lenses, etc. All of these data are freely accessible and everyone can then draw his own conclusions - probably a more interesting endeavour than just ranting about DXO's.

Every test is useful in its limited purpose. Ranting derives from 2 major causes:
- Incapability of understanding the test methodology, and thus both its usefulness and limitations: this leads to labeling as stupid.
- Results are not what one wants to hear: this leads to discrediting the tests, claiming bias and second interests, etc etc.

I agree with those who thinks that, if it was the other way around, this forum would be full of DXO ambassadors. Same as the recent "conversion" of Scott Kelby to Canon has not raised any suspicions about the actual circumstances as they were presented; it has been a genuine event of a prodigal son finally seeing the true light.

That post was a mini-festival of misinformation...

The key points are:

  • DxO does not disclose their methods for deriving their scores, which renders their scores meaningless
  • Some of DxO's measurements have errors, which makes all of their measurements suspect
  • DxO does not acknowledge their mistakes and issue corrigenda, but rather silently modify the original data, rendering their scientific ethics questionable

I'm not saying their information is useless, but I do suggest people view their measurements with caution, and ignore their Scores.

By the way, your recollection of facts is suspect, too.  "A genuine event of a prodigal son finally seeing the true light?"  Go back and re-read the Kelby conversion thread(s)...there was pretty broad support for the idea that he did it for the money.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #36 on: February 08, 2014, 09:06:15 AM »

Don Haines

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2806
  • Posting cat pictures on the internet since 1986
    • View Profile
Re: Why the DxO bashing?
« Reply #37 on: February 08, 2014, 09:14:49 AM »
By the way, your recollection of facts is suspect, too.  "A genuine event of a prodigal son finally seeing the true light?"  Go back and re-read the Kelby conversion thread(s)...there was pretty broad support for the idea that he did it for the money.
NO!!!!!!! Tell me it isn't so!!!!! Come on now, who ever heard of a business doing something for money! :)
The best camera is the one in your hands

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Why the DxO bashing?
« Reply #38 on: February 08, 2014, 09:18:54 AM »

The key points are:

  • DxO does not disclose their methods for deriving their scores, which renders their scores meaningless
  • Some of DxO's measurements have errors, which makes all of their measurements suspect
  • DxO does not acknowledge their mistakes and issue corrigenda, but rather silently modify the original data, rendering their scientific ethics questionable


On which I totally agree. In fact I have specifically addressed the difference between data themselves and their interpretation, as well as pointing out the need of accepting the limitation of any single test and understanding the useful information it provides without labeling the whole thing as pointless, biased or false.

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2678
    • View Profile
Re: Why the DxO bashing?
« Reply #39 on: February 08, 2014, 10:06:26 AM »
hahaha... it is funny to read.

I'll repeat what I said before but make it briefer:

If Canon's cameras had of been king of the hill on DxO and not Nikon/Sony then you can bet that people here would be saying DxO is right and how good they are and Nikon/Sony people would have been whinging like you see people here whinging.

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2678
    • View Profile
Re: Why the DxO bashing?
« Reply #40 on: February 08, 2014, 10:12:51 AM »
DXO pretends these differences are significant because if they tested bodies and lenses and consistently wrote that for 99% of circumstances, you can't tell the difference. Well...that wouldn't help them sell their software or drive people to their website or give them free publicity through photography forums.

I don't think the published results help them sell their software in so much as it is a by product of the work that they do to enable their software to do what it does and DxO Optics Pro is without peer.

Quote
Similarly, camera brand partisans magnify the importance of these small differences because it confirms their biases, regardless of which side they are on. Canon fans see it as confirmation that DXO is biased, Nikon fans see it as confirmation that their brand is better. The reality is that unless your primary subject is test charts, almost none of this matters.

+1

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2678
    • View Profile
Re: Why the DxO bashing?
« Reply #41 on: February 08, 2014, 10:24:10 AM »
...
Many Canon Rumors trolls may experience great satisfaction from demonstrating that ~2 stop deficit in low ISO DR by shooting images with the lens cap on, then pushing those images 4-5 stops in post. 
...
Many times in the DR debate, those bashing Canon sensors have been asked to provide examples of shots ruined by Canon's 'poor low ISO DR' that would have been saved by those extra two stops.  Personally, I have almost no examples of that situation - in many scenes, the ~12 stops I get is sufficient, and when the scene DR is greater than 12 stops, it's almost always greater than 14 stops, too.

Dude, it's not just the extra DR but the rampant noise in the shadow details of Canon files. This all but makes the bottom stop or two useless.

To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

So to mimic what has been said about the dynamic range of the 5D Mark II (and by extension at least that of the 5D Mark III) "usable DR of the camera is one or two stops lower than what is measured due to the inability to use the shadows."

Orangutan

  • 6D
  • *****
  • Posts: 576
    • View Profile
Re: Why the DxO bashing?
« Reply #42 on: February 08, 2014, 11:04:50 AM »
To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

Would you please post that raw file?

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #42 on: February 08, 2014, 11:04:50 AM »

David Hull

  • EOS M2
  • ****
  • Posts: 187
    • View Profile
Re: Why the DxO bashing?
« Reply #43 on: February 08, 2014, 11:32:45 AM »
If nothing else comes out of this discussion, at least we can hope that the folks at DxO will be reading this, too.

Nah.... it's not in French.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13586
    • View Profile
Re: Why the DxO bashing?
« Reply #44 on: February 08, 2014, 12:11:31 PM »
To give you an example of a scene I tried (and failed) to capture with Canon equipment, I had a setting sun behind a building that I could see through the door on the east, down a corridor and out the open door on the west side. As you can imagine, the detail outside the building on the west side was brightly lit (direct/diffuse sunlight), the side of the building I was on was maybe in the 50% grey area and the interior of the building was quite dark. There was very limited ability to expose to the right due to the outdoor area being lit by the sun but at the same time, if I didn't push it then the interior was lost to noise from Canon's sensor.

From your description, the scene would likely have had >15 stops of DR, supporting my earlier statement.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #44 on: February 08, 2014, 12:11:31 PM »