In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?
Why the DxO bashing?
- Thread starter JumboShrimp
- Start date
I don't even waste my time on this DXo crap... they're all Nikon fanboys and shizzle!
In the past, I've seen so many ridiculous reviews about sharpness and sensor quality in their website... hahahah, what a joke!
samples of that crap below:
http://fstoppers.com/dxomark-rates-canon-1d-x-worse-than-the-three-year-old-nikon-d3s
http://nikonrumors.com/2013/07/25/new-dxomark-king-the-nikon-200mm-f2g-ed-vr-ii-is-the-sharpest-lens-ever-tested.aspx/ (everybody knows the Canon 300 2.8 II IS is the sharpest lens ever produced)
http://photo.net/canon-eos-digital-camera-forum/00aBCm
http://fakechuckwestfall.wordpress.com/2012/05/14/5d-mark-iii-d800-d4-dxomark-sucks-balls/
No, sorry.. not for me!
In the past, I've seen so many ridiculous reviews about sharpness and sensor quality in their website... hahahah, what a joke!
samples of that crap below:
http://fstoppers.com/dxomark-rates-canon-1d-x-worse-than-the-three-year-old-nikon-d3s
http://nikonrumors.com/2013/07/25/new-dxomark-king-the-nikon-200mm-f2g-ed-vr-ii-is-the-sharpest-lens-ever-tested.aspx/ (everybody knows the Canon 300 2.8 II IS is the sharpest lens ever produced)
http://photo.net/canon-eos-digital-camera-forum/00aBCm
http://fakechuckwestfall.wordpress.com/2012/05/14/5d-mark-iii-d800-d4-dxomark-sucks-balls/
No, sorry.. not for me!
Upvote
0
JumboShrimp said:
For the most part, it is what people do with the DxO results, their interpretations of them and the conclusions they draw from them that are the problem. The fundamental measurements are good but to try to roll that all into one single summary number is problematic.
Upvote
0
Much of the time, their data is accurate, but many disagree with their lens testing methods, which appear to be a matter of someone's opinion but with no justification.
Where the big complaint comes in is in their assigning a numerical rating to a camera or a lens and refusing to tell us how they arrived at that number. Many times, the data may be the same, or even better for a lens, but it is rated lower than the competition. Why? They have been called out on this, and, in some cases, it was so blatant that they changed the rating.
When testing cameras, they only look at the sensor, and their rating reflects its performance in bright lighting at low ISO. A camera may fall down at high ISO, but that does not play into their rating. They down size the images to compare them. However, many want a high MP camera, not a 8mp camera. They may want to crop images. Their data or ratings don't tell you that there is a severe issue cropping a 36mp camera to 8mp because the individual pixels are very noisy.
That numerical value is used by people who do not understand the source (Its a secret), and then they buy a product and may find out it doesn't perform for their use. Others who use the product differently may love it.
If you read their test methods and realize the method used to conduct the test, you can then form your own opinion. Often, I find my opinion to be different.
Where the big complaint comes in is in their assigning a numerical rating to a camera or a lens and refusing to tell us how they arrived at that number. Many times, the data may be the same, or even better for a lens, but it is rated lower than the competition. Why? They have been called out on this, and, in some cases, it was so blatant that they changed the rating.
When testing cameras, they only look at the sensor, and their rating reflects its performance in bright lighting at low ISO. A camera may fall down at high ISO, but that does not play into their rating. They down size the images to compare them. However, many want a high MP camera, not a 8mp camera. They may want to crop images. Their data or ratings don't tell you that there is a severe issue cropping a 36mp camera to 8mp because the individual pixels are very noisy.
That numerical value is used by people who do not understand the source (Its a secret), and then they buy a product and may find out it doesn't perform for their use. Others who use the product differently may love it.
If you read their test methods and realize the method used to conduct the test, you can then form your own opinion. Often, I find my opinion to be different.
Upvote
0
L
Loswr
Guest
Their Scores are biased, in both disclosed and undisclosed ways. Their Sensor Score is weighted toward ISO 100 (2 of 3 metrics are used only at ISO 100 despite being measured throughout the range), and they state the overall score is a 'weighted average' of the three subscores, but don't reveal the weighting. Their Lens Score is based on performance in 150 lux illumination (like a dimly lit warehouse), so a lens will score higher when tested on a body with better high-ISO performance (so, how is it a 'lens score'?); similarly, when comparing two lenses, a lens that's worse on all the optical measures (sharpness, CA, etc.) can get a higher Score than an optically superior lens, based on the bodies on which they're tested, again based on that 150 lux bias. That bias also means transmission is disproportionately weighted - the 50/1.8 II gets a higher Score than the 600/4L IS II for that reason.
They lost a lot of credibility when they tested the Canon 70-200/2.8L IS II and concluded the original/MkI version was better - that disagreed with everyone who'd used or tested both, and when called on that, they said there was no mistake. But, about a year later they quietly updated their tests of the MkII and now it shows better performance than the original. I suspect they've also botched the testing of the 17-40L - they 'show' it to be just about as sharp in the extreme corners as the center wide open (it's mush in the corners at f/4), and wide open it shows as sharper than the 16-35L II stopped down to f/8 (totally false).
With the exception of errors like the above, their Measurements are useful. But their Scores are biased (so I call them Biased Scores = BS, aka bovine scat). For sensors, they're not applicable across the range of uses, and for lenses the Scores aren't even mainly based on the measurements.
A secondary issue is that review/comparison sites like Snapsort use the DxOMark Sensor Scores, without linking to the underlying measurements.
They lost a lot of credibility when they tested the Canon 70-200/2.8L IS II and concluded the original/MkI version was better - that disagreed with everyone who'd used or tested both, and when called on that, they said there was no mistake. But, about a year later they quietly updated their tests of the MkII and now it shows better performance than the original. I suspect they've also botched the testing of the 17-40L - they 'show' it to be just about as sharp in the extreme corners as the center wide open (it's mush in the corners at f/4), and wide open it shows as sharper than the 16-35L II stopped down to f/8 (totally false).
With the exception of errors like the above, their Measurements are useful. But their Scores are biased (so I call them Biased Scores = BS, aka bovine scat). For sensors, they're not applicable across the range of uses, and for lenses the Scores aren't even mainly based on the measurements.
A secondary issue is that review/comparison sites like Snapsort use the DxOMark Sensor Scores, without linking to the underlying measurements.
Upvote
0
Any attempt to reduce a complex system used under diverse conditions for diverse goals to a single number is doomed to failure.
Upvote
0
rpt said:
I love to tease Neuro that my 50F1.8 is better than his 600F4 because DXO rates it higher.... Regardless of how the ratings are determined, the entire concept is doomed to failure. You can not represent the superiority of a lens with a single number. There are so many factors which can't be rated this way, including "what are you going to use it for". If I want to take pictures of a eagle flying above the trees, the 600F4 is the way to go... but if I want to take a picture of Fluffy sleeping on my legs, it's a terrible lens... and then there is affordability, a highly personal criteria which negates all the technical merits because if you can't afford to buy it, does it really matter how good it is? Some criteria vary among the individual from day to day, such as weight... I would have no problem lugging an 800F5.6 around the local conservation area, but there is no way I am going to lug it for a two week backpacking trip in the mountains....
Upvote
0
Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch.
As for DxOmark's single numerical score, mostly useless.
Their raw measurements, however, provide a lot of useful and easy to compare information on sensor performance. As do their lens data, just do your own comparisons.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch.
As for DxOmark's single numerical score, mostly useless.
Their raw measurements, however, provide a lot of useful and easy to compare information on sensor performance. As do their lens data, just do your own comparisons.
Upvote
0
I cannot help but question the measurement methodology of a group who (either fraudulently or ignorantly) uses biased and misleading summary statistics to put forth claims about camera/lens performance. That is to say, if you don't analyze your data properly but staunchly claim to be fair and objective, then it is my obligation to question your data collection methods as well, because your entire process is now suspect. That is what any good scientist does.
Upvote
0
WOW! ... if DxO said that, it must be true ;D ... I am going to buy the EF 50 f/1.8 II and frame it in a fancy glass case and chant praises, worshiping its higher state of being daily ;D ... maybe it could lead to the birth of a new cult called the DxO worshipers ... unfortunately, I might be a bit too late, coz I'm told that something like that already exists ;Dneuroanatomist said:
Upvote
0
What?rpt said:Nope! An ordinary engineer like meOrangutan said:
... he is just an ordinary engineer and does not even have the highly acclaimed Masters in Bull$h!tting Arts? ... how dare he! ;D
Upvote
0
L
Loswr
Guest
Aglet said:Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch.
Many Canon Rumors trolls may experience great satisfaction from demonstrating that ~2 stop deficit in low ISO DR by shooting images with the lens cap on, then pushing those images 4-5 stops in post.
The majority of Canon devotees, at least here on CR, seem to be fairly objective about the issue. It's widely acknowledged that Canon sensors deliver less low ISO DR than Nikon/Sony sensors. The fact remains that people buy cameras, not bare silicon sensors. If low ISO DR is someone's only criterion for judging a camera's performance (and for a very tiny minority of people, that may be the case), they should choose something other than a Canon camera. But for most people, what matters is the performance of the system as a whole (camera + lenses + flashes + accessories), and that's where Canon usually wins. As Don Haines is fond of saying, who cares how many stops of dynamic range a blurry picture has?
Many times in the DR debate, those bashing Canon sensors have been asked to provide examples of shots ruined by Canon's 'poor low ISO DR' that would have been saved by those extra two stops. Personally, I have almost no examples of that situation - in many scenes, the ~12 stops I get is sufficient, and when the scene DR is greater than 12 stops, it's almost always greater than 14 stops, too.
But for those who would like to persist in this debate, I have found a relevant example showing how an extra two stops of DR can keep the sunlit outdoors from blowing out when shooting an indoor candlelit scene...
Attachments
Upvote
0
L
Loswr
Guest
chromophore said:
DxO has the phrase "Image Science" as part of their logo, but their practices aren't consistent with that phrase. I'm most concerned by the 'black box' calculation for their summary Sensor Score and Lens Score (methods should be published), and by the fact that they released data which was incorrect, defended it, then subsequently changed it with no acknowledgement of their error. Also, I'm noticing that the more I delve into their Measurements, the more I find errors (for example, I just looked at the Canon 28-300L measurements and their actuance data shown visually as field maps are ~10% lower relative to the same data plotted on a graph as a profile).
Upvote
0
In my more cynical moments, I imagine that if DXO gave more glowing reviews of Canon lenses, the opinions of many people on this forum might change. ;D
Upvote
0
unfocused
Photos/Photo Book Reviews: www.thecuriouseye.com
AcutancePhotography said:
Probably true. But I have a slightly different perspective.
Whenever I've looked at DXO results, I've been struck by how small the differences are that they are attempting to measure and quantify.
DXO pretends these differences are significant because if they tested bodies and lenses and consistently wrote that for 99% of circumstances, you can't tell the difference. Well...that wouldn't help them sell their software or drive people to their website or give them free publicity through photography forums.
Similarly, camera brand partisans magnify the importance of these small differences because it confirms their biases, regardless of which side they are on. Canon fans see it as confirmation that DXO is biased, Nikon fans see it as confirmation that their brand is better. The reality is that unless your primary subject is test charts, almost none of this matters.
Upvote
0