DXO: Fair and Balanced

[Don Haines]

Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?

There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are doomed.

Yes, but imagine if the food is all rated from 1 to 100.... Does this mean that you would be better off ordering the 87 rated pancakes than the 79 rated pie? Even though you really wanted the 73 rated cup of coffee?

 

[Orangutan]

Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?

There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are doomed.

It the restaurant critic's motto, Food Science? :

+1

This is the point I made above: if this is done "for fun" (as marketing) then who cares? The problem is that DxO is promoting this as objective.

If you have a single restaurant critic go into a restaurant (presumably in disguise), then it's entertainment.

If you have a blind test-test using 100 graduates of a culinary academy, you can begin to call it science.

 

[ahsanford]

I will be captain obvious. They compile a lot of specific data and then diminish its usefulness by combining it all into one number.

For instance. The sharpness value is an average across the field at all apertures. Maybe that's important to know for some reason? I don't care how sharp it is stopped all the way down, I don't use lenses that way.

This is why the 35 Art was considered sharper than the 35L II on the 5D3. The 35L II goes to f/22 while the 35 Art only stops down to f/16. The poor sharpness value of f/22 pulled down the average of the 35L II, which is nonsense.

- A

 

[PhotographyFirst]

I think DXO is quite genius and deliberate with their scoring system.

The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!"

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores. Are most of those people just visiting to get numbers for arguing or are most of them actually using DXO to make a purchase choice? I would think that someone who is already in a camera system is purchasing their gear based on a need rather than just trying to find the highest scoring lens to purchase regardless of being useful for their work. I don't think many people are trying to decide between a 50mm prime and a 600mm prime based on which one scores higher.

All of the photo rumors and gear websites know very well that DXO is prime clickbait for readers. All they have to do is post what DXO does and they get their clicks regardless of how controversial the DXO content really is. Much better than having to post a link to a complex review with test shots, charts, graphs, and other more useful data.

Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.

 

[3kramd5]

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores.

I don't see how they possibly could unless they are exceedingly heavy-handed with tracking cookies.

 

[bdunbar79]

I think DXO is quite genius and deliberate with their scoring system.

The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!"

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores. Are most of those people just visiting to get numbers for arguing or are most of them actually using DXO to make a purchase choice? I would think that someone who is already in a camera system is purchasing their gear based on a need rather than just trying to find the highest scoring lens to purchase regardless of being useful for their work. I don't think many people are trying to decide between a 50mm prime and a 600mm prime based on which one scores higher.

All of the photo rumors and gear websites know very well that DXO is prime clickbait for readers. All they have to do is post what DXO does and they get their clicks regardless of how controversial the DXO content really is. Much better than having to post a link to a complex review with test shots, charts, graphs, and other more useful data.

Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.

Great post! LOL!

Sales? No need. DxOmark consistently scores all of Nikon's gear higher than Canon's, year after year after year, yet Nikon gains no market share on Canon in the camera market. Love it!

 

[Loswr]

Factual errors and a refusal to admit mistakes, combined with an obvious bias in conclusions. Dilbert and DxOMark have much in common.

 

[Don Haines]

...
Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.

Why does bigger have to mean better? Or is this just another male failing? (Since you mention "males"...)

If Canon produced a 50mm lens that was perfect, T-stop was the same as F-stop, no CA, equal sharpness middle to edge ... why shouldn't that get a score higher than any other lens, even if it only cost (say) $250?

On the other end of the scale, if Canon produced a 600mm lens that was built like a tank, could fall 3 stories and still work perfectly but had bad CA, wasn't sharp corner to corner, T-stop was 2/3s slower than F-stop and cost $5000, why should it score highly?

Similarly, just how good is that 600mm lens going to be in a portrait studio?

If you waltzed into a studio with a 600mm lens and waved that about to show how superior you were to the photographer using a 85/1.8 to shoot customers you'd look just as silly as with the 50/STM at football.

DxO measure the ability of the lens to deliver optical results. That's it. It's then up to you to choose which lens to use for a particular assignment. Knowing the DxO scores for various 600mm lenses lets you choose which one is best for you compared to price. The scoring mechanism also gives you an indication of the relative image quality that will be captured by the camera.

The DXO scores are extremely heavily biased towards T-stop values....so much so, that realistically, that is the only metric that counts.

The problem that most people have with DXO is that their rating system is based on unrealistic values and is kept secret. This is not an open and unbiased system.

As a researcher, when I run an experiment I have to show the exact setup of the equipment, show all the settings, include the calibration data for the equipment ( even the serial numbers), and all the data. Good runs and bad runs are included. multiple runs for repeatability..... And in the end, when I write up the report, different conclusions for different conditions.... If I were to say that "this radio is the best for everything because it handled this one unrealistic scenario the best", I would be encouraged to either take a long vacation or to retire.....

 

[3kramd5]

I think DXO is quite genius and deliberate with their scoring system.

The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!"

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores. Are most of those people just visiting to get numbers for arguing or are most of them actually using DXO to make a purchase choice? I would think that someone who is already in a camera system is purchasing their gear based on a need rather than just trying to find the highest scoring lens to purchase regardless of being useful for their work. I don't think many people are trying to decide between a 50mm prime and a 600mm prime based on which one scores higher.

All of the photo rumors and gear websites know very well that DXO is prime clickbait for readers. All they have to do is post what DXO does and they get their clicks regardless of how controversial the DXO content really is. Much better than having to post a link to a complex review with test shots, charts, graphs, and other more useful data.

Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.

Why does bigger have to mean better?

Am I blind? I'm failing to see an reference to size in the quoted post.

 

[dolina]

I like DXO's Mobile review. Got me to consider the Nexus 6P.

 

[Larsskv]

...
The DXO scores are extremely heavily biased towards T-stop values....so much so, that realistically, that is the only metric that counts.
...

Is it simply "lower T-stop is better" or "T-stop closer to F-stop is better"?

How would anybody outside DXO know?

I suspect it is lower t stop=better, because small aperture lenses suffer so much in DXO tests. If tests are done at 150 lux, 1/60 sek and base ISO, lower t stop will be better.

 

[3kramd5]

I think DXO is quite genius and deliberate with their scoring system.

The male gender sure has a strange willingness to argue over anything and everything. Just like with sports, audio gear, computers, video game systems, cars, and so on. Having numbers to argue over regardless of how meaningless and wrong those numbers might be, is like fuel for the flames. Even better when those numbers are dumbed down as much as possible. "7 is better than 6.9! Your choice sucks! You loser of purchasing crappy low scoring gear!"

I would be very curious to see if DXO has calculated the number of sales conversions they get when people visit to see DXO scores. Are most of those people just visiting to get numbers for arguing or are most of them actually using DXO to make a purchase choice? I would think that someone who is already in a camera system is purchasing their gear based on a need rather than just trying to find the highest scoring lens to purchase regardless of being useful for their work. I don't think many people are trying to decide between a 50mm prime and a 600mm prime based on which one scores higher.

All of the photo rumors and gear websites know very well that DXO is prime clickbait for readers. All they have to do is post what DXO does and they get their clicks regardless of how controversial the DXO content really is. Much better than having to post a link to a complex review with test shots, charts, graphs, and other more useful data.

Next time I am at a sporting event, I am going to sneak down to the sidelines with a Sony A7s II, the Canon 50mm STM, and a printout of the DXO scores. Those guys using the 600 II lens and 1dx will be full of regret when they see my lens is several points better than theirs and my camera body totally destroys them for sports shooting scores.

Why does bigger have to mean better?

Am I blind? I'm failing to see an reference to size in the quoted post.

Apologies if you missed the 50s and 600s, I've highlighted them for your benefit.

You are implying that the poster thinks the 600 is better than the 50 because it's bigger than the 50.

Rather, he was CLEARLY (see red) talking about applicability to the given subject matter, which is the major failing of a composite scoring system applied to items with a wide range of intended uses.

Size is irrelevant.

 

[Maiaibing]

Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to?

Because no one demands you buy their products or spend your time finding their site and reading the info they put on the web for you to enjoy/hate for free.

Lots of companies claim science backs their products and statements without any such disclosure. Cosmetics companies are a case in point. Methodology and measurements is often - correctly - considered a valuable intellectual property that should be protected. No surprise here.

 

[bdunbar79]

Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to?

Because no one demands you buy their products or spend your time finding their site and reading the info they put on the web for you to enjoy/hate for free.

Lots of companies claim science backs their products and statements without any such disclosure. Cosmetics companies are a case in point. Methodology and measurements is often - correctly - considered a valuable intellectual property that should be protected. No surprise here.

No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.

Anytime you publish a journal article, or file a patent, etc.

The other point, is that everything you just typed doesn't make DxOmark scientific. My only point was that they are not scientific...because they are not. Not by a long shot. So why claim to be?

 

[Loswr]

The scoring system is fine. There is nothing wrong with the scoring system.

There is nothing wrong with measuring weight in stone or length in furlongs, but it's not too useful for the vast majority of people.

There is something wrong with a scoring system that is biased, undisclosed, and may be determined differently for different products.

 

[Don Haines]

The scoring system is fine. There is nothing wrong with the scoring system.

The scoring system rates the T-stop value drastically higher than any other optical properties of the lens....

If this is fine with you, then so be it.... but many others seem to disagree.

My problem with DXO is the entire concept of reducing things to a single number is deeply flawed. No matter how they arrive at that number, no matter what their magic formula is, the concept of a single number is deeply flawed. They would have been better off to have stopped at the sub-scores.

 

[Maiaibing]

The scoring system is fine. There is nothing wrong with the scoring system.

My problem with DXO is the entire concept of reducing things to a single number is deeply flawed.

While I understand all reviews at DpReview etc. etc. etc. are imminently painful to you, single number scores - sometimes much less developed than DXO's - enjoy extremely wide spread popularity on the net, in newspapers, magazines etc.

Why single DXO out for this?

 

[Maiaibing]

No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.

I hope this is not a desperate attempt to turn around from a bad argument by drawing a strawman. Safety testing has nothing at all to do with their claims of scientific testing supporting the value of their products.

This is what you were claiming was wrong of DXO to do:

"Experience the first creme with IntuiGen Technology™ that knows your skin's needs.
This high-performance, multi-action moisturizer dramatically reduces the look of multiple signs of aging: lines, wrinkles, loss of firmness, dullness and dehydration.
Breakthrough IntuiGen Technology™ helps activate skin's own revitalization. It specifically addresses your unique anti-aging needs all at once, revealing a younger and more beautiful look.
The lightweight, silky soft creme feels like no other, cushioning your skin with sensuously rich, refreshing moisture.
Tested and proven: Instantly, skin looks smoother, clearer, more radiant. In just 4 weeks, 92% of women showed significantly firmer skin.


I suggest you contact Estee Lauder and ask them for disclosure.

 

[ahsanford]

Why single DXO out for this?

Numerical metrics/ratings make sense for individual aspects of a product's performance, for instance:

• I am curious to see if lens X does well making the jump from a 5D3 to a 5DS.
• I am curious to know how much light a lens really lets in.
• Sensor DR, high ISO performance and other metrics can be measured and reported

Skipping Neuro's (absolutely legitimate) beef with DXO's individual test methods, I'm not opposed to measuring things like sharpness, low light sensitivity, etc. and giving it a number.

Where I go Defcon 1 is how DXO rolls things up:

[list type=decimal]
[*]Roll those individual metrics together into a nonsensical, inconsistent overall score that seems to prioritize minutiae like Transmission (or even the outright max aperture) over the more bread and butter considerations of sharpness, aberrations, vignetting, etc.
[*]Overwhelmingly weight the use of that overall score to peg if something is 'ground-breaking' or 'disappointing'
[*]Apply a noogie to other products without considering usage, photographic need, features, etc. like some sterile United Nations of optical quality.
[/list]

Again, if DXO simply did away with the Overall Score, people could go there and mine the data they needed when considering new gear. That's all they need to do.

- A

 

[bdunbar79]

No. Cosmetic companies have to do regulated toxicological testing, where everything, every little single thing, MUST be recorded and disclosed, and audited by a regulating body.

I hope this is not a desperate attempt to turn around from a bad argument by drawing a strawman. Safety testing has nothing at all to do with their claims of scientific testing supporting the value of their products.

This is what you were claiming was wrong of DXO to do:

"Experience the first creme with IntuiGen Technology™ that knows your skin's needs.
This high-performance, multi-action moisturizer dramatically reduces the look of multiple signs of aging: lines, wrinkles, loss of firmness, dullness and dehydration.
Breakthrough IntuiGen Technology™ helps activate skin's own revitalization. It specifically addresses your unique anti-aging needs all at once, revealing a younger and more beautiful look.
The lightweight, silky soft creme feels like no other, cushioning your skin with sensuously rich, refreshing moisture.
Tested and proven: Instantly, skin looks smoother, clearer, more radiant. In just 4 weeks, 92% of women showed significantly firmer skin.


I suggest you contact Estee Lauder and ask them for disclosure.

I'm sorry, but where in their claim does it ever say "science" or "scientific"? Nowhere, because they are not allowed to.

Things starting to click now?

If you have a therapeutic cream, then they must do efficacy testing to make scientific claims, but that's more pharmaceutical.

But anyways, I guess since cosmetic companies mislead with garbage science, it's okay for DxOmark to do so. So in a weird way, by using your example, you're admitting DxOmark is garbage science, and also okay with that...because afterall, the cosmetic industry does it.

Unfortunately, cosmetic companies have to make misleading claims because this is what consumers respond to.

Just like DxOmark...