July 29, 2014, 03:59:01 PM

Author Topic: Why the DxO bashing?  (Read 20835 times)

JumboShrimp

  • Rebel T5i
  • ****
  • Posts: 107
    • View Profile
Why the DxO bashing?
« on: February 06, 2014, 07:42:29 PM »
In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?

canon rumors FORUM

Why the DxO bashing?
« on: February 06, 2014, 07:42:29 PM »

hemidesign

  • PowerShot G1 X II
  • ***
  • Posts: 74
    • View Profile
Re: Why the DxO bashing?
« Reply #1 on: February 06, 2014, 08:26:23 PM »
I don't even waste my time on this DXo crap... they're all Nikon fanboys and S___!
In the past, I've seen so many ridiculous reviews about sharpness and sensor quality in their website... hahahah, what a joke!

samples of that crap below:

http://fstoppers.com/dxomark-rates-canon-1d-x-worse-than-the-three-year-old-nikon-d3s

http://nikonrumors.com/2013/07/25/new-dxomark-king-the-nikon-200mm-f2g-ed-vr-ii-is-the-sharpest-lens-ever-tested.aspx/ (everybody knows the Canon 300 2.8 II IS is the sharpest lens ever produced)

http://photo.net/canon-eos-digital-camera-forum/00aBCm

http://fakechuckwestfall.wordpress.com/2012/05/14/5d-mark-iii-d800-d4-dxomark-sucks-balls/





No, sorry.. not for me!  >:(

Orangutan

  • 6D
  • *****
  • Posts: 580
    • View Profile
Re: Why the DxO bashing?
« Reply #2 on: February 06, 2014, 08:27:01 PM »
In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?

Neuro, I believe that's your cue...   :P

David Hull

  • EOS M2
  • ****
  • Posts: 187
    • View Profile
Re: Why the DxO bashing?
« Reply #3 on: February 06, 2014, 08:27:56 PM »
In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?

For the most part, it is what people do with the DxO results, their interpretations of them and the conclusions they draw from them that are the problem.  The fundamental measurements are good but to try to roll that all into one single summary number is problematic.

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8272
    • View Profile
Re: Why the DxO bashing?
« Reply #4 on: February 06, 2014, 08:36:00 PM »
Much of the time, their data is accurate, but many disagree with their lens testing methods, which appear to be a matter of someone's opinion but with no justification.
 
Where the big complaint comes in is in their assigning a numerical rating to a camera or a lens and refusing to tell us how they arrived at that number.  Many times, the data may be the same, or even better for a lens, but it is rated lower than the competition.  Why?  They have been called out on this, and, in some cases, it was so blatant that they changed the rating.
 
When testing cameras, they only look at the sensor, and their rating reflects its performance in bright lighting at low ISO.  A camera may fall down at high ISO, but that does not play into their rating.  They down size the images to compare them.  However, many want a high MP camera, not a 8mp camera.  They may want to crop images.  Their data or ratings don't tell you that there is a severe issue cropping a 36mp camera to 8mp because the individual pixels are very noisy.
 
That numerical value is used by people who do not understand the source (Its a secret), and then they buy a product and may find out it doesn't perform for their use.  Others who use the product differently may love it.
If you read their test methods and realize the method used to conduct the test, you can then form your own opinion.  Often, I find my opinion to be different.
 
 

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13596
    • View Profile
Re: Why the DxO bashing?
« Reply #5 on: February 06, 2014, 08:49:21 PM »
Their Scores are biased, in both disclosed and undisclosed ways.  Their Sensor Score is weighted toward ISO 100 (2 of 3 metrics are used only at ISO 100 despite being measured throughout the range), and they state the overall score is a 'weighted average' of the three subscores, but don't reveal the weighting.  Their Lens Score is based on performance in 150 lux illumination (like a dimly lit warehouse), so a lens will score higher when tested on a body with better high-ISO performance (so, how is it a 'lens score'?); similarly, when comparing two lenses, a lens that's worse on all the optical measures (sharpness, CA, etc.) can get a higher Score than an optically superior lens, based on the bodies on which they're tested, again based on that 150 lux bias. That bias also means transmission is disproportionately weighted - the 50/1.8 II gets a higher Score than the 600/4L IS II for that reason.

They lost a lot of credibility when they tested the Canon 70-200/2.8L IS II and concluded the original/MkI version was better - that disagreed with everyone who'd used or tested both, and when called on that, they said there was no mistake.  But, about a year later they quietly updated their tests of the MkII and now it shows better performance than the original.  I suspect they've also botched the testing of the 17-40L - they 'show' it to be just about as sharp in the extreme corners as the center wide open (it's mush in the corners at f/4), and wide open it shows as sharper than the 16-35L II stopped down to f/8 (totally false).

With the exception of errors like the above, their Measurements are useful.  But their Scores are biased (so I call them Biased Scores = BS, aka bovine scat).  For sensors, they're not applicable across the range of uses, and for lenses the Scores aren't even mainly based on the measurements.

A secondary issue is that review/comparison sites like Snapsort use the DxOMark Sensor Scores, without linking to the underlying measurements.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Don Haines

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2815
  • Posting cat pictures on the internet since 1986
    • View Profile
Re: Why the DxO bashing?
« Reply #6 on: February 06, 2014, 09:03:52 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
The best camera is the one in your hands

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #6 on: February 06, 2014, 09:03:52 PM »

rpt

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2123
  • Could not wait for 7D2 so I got the 5D3
    • View Profile
Re: Why the DxO bashing?
« Reply #7 on: February 06, 2014, 09:33:59 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Don Haines

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2815
  • Posting cat pictures on the internet since 1986
    • View Profile
Re: Why the DxO bashing?
« Reply #8 on: February 06, 2014, 09:57:11 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

I love to tease Neuro that my 50F1.8 is better than his 600F4 because DXO rates it higher.... Regardless of how the ratings are determined, the entire concept is doomed to failure. You can not represent the superiority of a lens with a single number. There are so many factors which can't be rated this way, including "what are you going to use it for". If I want to take pictures of a eagle flying above the trees, the 600F4 is the way to go... but if I want to take a picture of Fluffy sleeping on my legs, it's a terrible lens... and then there is affordability, a highly personal criteria which negates all the technical merits because if you can't afford to buy it, does it really matter how good it is?  Some criteria vary among the individual from day to day, such as weight... I would have no problem lugging an 800F5.6 around the local conservation area, but there is no way I am going to lug it for a two week backpacking trip in the mountains....
The best camera is the one in your hands

Orangutan

  • 6D
  • *****
  • Posts: 580
    • View Profile
Re: Why the DxO bashing?
« Reply #9 on: February 06, 2014, 10:03:14 PM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Does he have an MBA, by chance?

Aglet

  • 1D Mark IV
  • ******
  • Posts: 908
    • View Profile
Re: Why the DxO bashing?
« Reply #10 on: February 06, 2014, 11:03:43 PM »
Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch. ;)

As for DxOmark's single numerical score, mostly useless.
Their raw measurements, however, provide a lot of useful and easy to compare information on sensor performance.  As do their lens data, just do your own comparisons.

chromophore

  • Power Shot G16
  • **
  • Posts: 23
    • View Profile
Re: Why the DxO bashing?
« Reply #11 on: February 07, 2014, 12:25:43 AM »
I cannot help but question the measurement methodology of a group who (either fraudulently or ignorantly) uses biased and misleading summary statistics to put forth claims about camera/lens performance.  That is to say, if you don't analyze your data properly but staunchly claim to be fair and objective, then it is my obligation to question your data collection methods as well, because your entire process is now suspect.  That is what any good scientist does.

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3321
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Why the DxO bashing?
« Reply #12 on: February 07, 2014, 01:07:59 AM »
Their Scores are biased, in both disclosed and undisclosed ways.  That bias also means transmission is disproportionately weighted - the 50/1.8 II gets a higher Score than the 600/4L IS II for that reason.
WOW! ... if DxO said that, it must be true  ;D ... I am going to buy the EF 50 f/1.8 II and frame it in a fancy glass case and chant praises, worshiping its higher state of being daily  ;D ... maybe it could lead to the birth of a new cult called the DxO worshipers ... unfortunately, I might be a bit too late, coz I'm told that something like that already exists ;D
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #12 on: February 07, 2014, 01:07:59 AM »

rpt

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2123
  • Could not wait for 7D2 so I got the 5D3
    • View Profile
Re: Why the DxO bashing?
« Reply #13 on: February 07, 2014, 01:56:44 AM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Does he have an MBA, by chance?
Nope! An ordinary engineer like me ;)

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3321
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Why the DxO bashing?
« Reply #14 on: February 07, 2014, 02:18:19 AM »
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Thanks! Lovely sentence. I am going to quote your words to a colleague. He wanted me to arrive at a single data quality number after analysing tens of columns in each of about five hundred or so database tables.

Does he have an MBA, by chance?
Nope! An ordinary engineer like me ;)
What? :o ... he is just an ordinary engineer and does not even have the highly acclaimed Masters in Bull$h!tting Arts? ... how dare he! ;D
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #14 on: February 07, 2014, 02:18:19 AM »