DXO: Fair and Balanced

[Orangutan]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.

 

[Neutral]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.

I do agree that better transparency could be beneficial for people who want to be confident that DXO measurements results are 100% correct.
On the other hand lack of such trasparency does not mean at all that their measurements are incorrect , wrong or fake.
Morever stating that are just assumptions not supported by any facts/data, just personal emotions.
I personally do not see any issues with the results of their measurements (not scores) looking at their data and doing my own tests. One interesting thing though that using best canon glass on my Sony a7rII I get perceptially better results than DXO publish for Canon 5DSR. They resolve to the every single pixel on a7rII. But this is after sharpness corrections in LR and a7rII sensor allows very agressive sharpening without any visual side effects.
Also DXO could be caught by other independant test labs if they have done something wrong.
Also it is not very difficult to do reversed calculations to prove that they are wrong in perceptual resolution measurements. Just use MTF50 of one known lens (e.g. sigma art 50 published at lensrental) and use DXO measurements data for this lens on Canon 5dsr as starting point to calculate lens resolution lp/mm and to compare with lens MTF50 results on lensrental. Also this would allow to estimate DXO measurements initial condition. This (reversed calculations) is common practice for proving that something correct or incorrect.
So I trust their data and have no bias or prejustice to them and this based on their history and my own evaluation of their measurements data.
DXO started long time back as company doing optical test/measurements with their DXO optical analyser and then later started with DXO Optic RAW converter which is now well recognized. They do all their lens/sensor measurements for DXO lens correction modules - for themselves - to provide best possible lens correction results in their SW. It would be very strange to think that they would be using wrong/fake measurements to get best results for their SW.
Having accumulated huge amount of measurement data they started to publish that for other people benefit and of course for marketing themselves.
More useful info for their DXO Analyzer and what they doing could be found here:
http://www.dxo.com/us/image-quality-evaluation/dxo-analyzer

 

[Orangutan]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.

I do agree that better transparency could be beneficial for people who want to be confident that DXO measurements results are 100% correct.
On the other hand lack of such trasparency does not mean at all that their meusurements are incorrect , wrong or fake.

You've made my point for me, it's a question of the burden of proof. If DxO is to be considered science or engineering, then the burden of proof rests fully on DxO to demonstrate, through transparency and supporting data, that all the work is correct. Otherwise, it is merely marketing, where the burden of proof shifts more toward the "you haven't proved it's wrong" end of the spectrum.

So long as DxO continues to obfuscate their methods, their work is marketing crap, and should not be given our respect.

Also DXO could be caught by other independant test labs if they have done something wrong.

On the other hand, we could just stop paying any attention at all to DxO's marketing crap, and read the independent tests instead.

 

[Neutral]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.

I do agree that better transparency could be beneficial for people who want to be confident that DXO measurements results are 100% correct.
On the other hand lack of such trasparency does not mean at all that their meusurements are incorrect , wrong or fake.

You've made my point for me, it's a question of the burden of proof. If DxO is to be considered science or engineering, then the burden of proof rests fully on DxO to demonstrate, through transparency and supporting data, that all the work is correct. Otherwise, it is merely marketing, where the burden of proof shifts more toward the "you haven't proved it's wrong" end of the spectrum.

So long as DxO continues to obfuscate their methods, their work is marketing crap, and should not be given our respect.

Also DXO could be caught by other independant test labs if they have done something wrong.

On the other hand, we could just stop paying any attention at all to DxO's marketing crap, and read the independent tests instead.

OK, then why not to ask officially ( kind of public petition ) their customers (including NASA, Samsung, Pentax, Panasonic, Sigma, FujiFilm etc.) to impose sanctions on DXO and stop using their measurement facilities/services crap. May be this would force DXO to publish all their measurements methods, conditions and calculations to general public ) Also to all users of DXO Optic to stop using this crappy product and return it back DXO )

 

[Orangutan]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

This is the key: DxO has insufficient transparency with regard to testing protocols and calculations. Without transparency, all inferences are meaningless. Furthermore, promoted inferences that lack supporting data should be considered marketing rather than science or engineering.

I do agree that better transparency could be beneficial for people who want to be confident that DXO measurements results are 100% correct.
On the other hand lack of such trasparency does not mean at all that their meusurements are incorrect , wrong or fake.

You've made my point for me, it's a question of the burden of proof. If DxO is to be considered science or engineering, then the burden of proof rests fully on DxO to demonstrate, through transparency and supporting data, that all the work is correct. Otherwise, it is merely marketing, where the burden of proof shifts more toward the "you haven't proved it's wrong" end of the spectrum.

So long as DxO continues to obfuscate their methods, their work is marketing crap, and should not be given our respect.

Also DXO could be caught by other independant test labs if they have done something wrong.

On the other hand, we could just stop paying any attention at all to DxO's marketing crap, and read the independent tests instead.

OK, then why not to ask officially ( kind of public petition ) their customers (including NASA, Samsung, Pentax, Panasonic, Sigma, FujiFilm etc.) to impose sanctions on DXO and stop using their measurement facilities/services crap. May be this would force DXO to publish all their measurements methods, conditions and calculations to general public ) Also to all users of DXO Optic to stop using this crappy product and return it back DXO )

Non sequitur, these are separate questions. What they do on contract is a matter between them and their customer. I'm referring to their publicly promoted material. There is no reason to believe they use equally rigorous methods for both, and some reason t believe they don't because they've been caught in errors several times. In short, their public database should be considered marketing material, not engineering data, except in those cases where they provide enough information to replicate their results, and where their testing is internally consistent.

 

[Larsskv]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

I personally do not see any issues with the results of their measurements (not scores) looking at their data and doing my own tests. One interesting thing though that using best canon glass on my Sony a7rII I get perceptially better results than DXO publish for Canon 5DSR.

Consider what you are writing here. You do not see any issues, but you get better "perceptually better results" with the A7RII than the 5DSR. Considering that extra resolution, that seems a little weird, don´t you think?

Did you compare apples to apples?

Did you take your test picture in a room with 150 lux, at 1/60 sec, at base ISO and pushed the file to correct lighting when you took your comparison picture? No? Because that is what DXO seems to do. I get "perceptually better results" with all my f/4 lenses, than what is reported by DXO. Why? Because I make a correct exposure in a sufficient amount of light, when I am shooting, and therefore DXO is irrelevant for my type of shooting, which I guess is quite common - at least with Canon shooters.. DXO is very misleading, and it´s hard to know when they don´t let you know how they rate their scores.

(Edit: I don´t see why my text is quoted here...) <fixed by mod>

 

[bdunbar79]

I guess as a scientist, I find them calling their measurements and scores "scientific" quite offensive. It's literally just wrong and it is not scientific. Whenever I submit my results, I HAVE to disclose ALL of my methodology and measurements, so why don't they have to? Well, because they're not doing real science because that is every part of real science. Disclose the scoring system and parameters and I'm fine, but until then, they are absolutely meaningless to the scientific community.

 

[Orangutan]

(Edit: I don´t see why my text is quoted here...)

You may be missing some quote tags somewhere. It's got someone else's words attributed to me.

 

[Larsskv]

(Edit: I don´t see why my text is quoted here...)

You may be missing some quote tags somewhere. It's got someone else's words attributed to me.

Your absolutely right! Sorry about that. I can´t recall exactly what I did. Maybe I deleted some quote tags, and the numbers of quote tags dont add up.

 

[Loswr]

I personally do not see any issues with the results of their measurements (not scores) looking at their data

Sometimes yes, there could be errors in measurements or records due to number of reasons.
Most outstanding case was with Canon EF 70-200 f/2.8L IS USM II when DXO reported lower perceptial resolution compared to the previous generation, did not publicully admitted that this was wrong but eventially corrected measurements results.

Perhaps you don't have any problem with unethical behavior, but when an organization purports to be scientifically-based but fails to fully disclose their methods, they are not acting in accordance with the established practice of science. Moreover, when they are made aware of an error and defend their data rather than publicly acknowledging their error, then subsequently alter their underlying data but fail to correct their conclusions based on those faulty data, that behavior is counter to the ethical principles to which scientists are expected to adhere.

DxO Image Science...Bad Science...Biased Scores...BS.

 

[Neutral]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

I personally do not see any issues with the results of their measurements (not scores) looking at their data and doing my own tests. One interesting thing though that using best canon glass on my Sony a7rII I get perceptially better results than DXO publish for Canon 5DSR.

Consider what you are writing here. You do not see any issues, but you get better "perceptually better results" with the A7RII than the 5DSR. Considering that extra resolution, that seems a little weird, don´t you think?

Did you compare apples to apples?

Did you take your test picture in a room with 150 lux, at 1/60 sec, at base ISO and pushed the file to correct lighting when you took your comparison picture? No? Because that is what DXO seems to do. I get "perceptually better results" with all my f/4 lenses, than what is reported by DXO. Why? Because I make a correct exposure in a sufficient amount of light, when I am shooting, and therefore DXO is irrelevant for my type of shooting, which I guess is quite common - at least with Canon shooters.. DXO is very misleading, and it´s hard to know when they don´t let you know how they rate their scores.

(Edit: I don´t see why my text is quoted here...)

[

I agree that using this combination of ISO, shuter speed and luminance creates a bit difficult overall setup and gives latest better tech sensors some advantage.
With higher luminance results could be better.
And this applies in particular to the a7rII sensor. As I mentioned I can push sharpness very agressively using 0.5 radius without seeing any side effects. As results on final processed image I can see that my EF24-70 f/2.8L IS USM II resolve to every single pixel on a7rII. The same for the latest 70-200 and 100-400 .
So better visual results is contribution of both the lens quality and sensor quality and indicator of this combo performace.

On the other hand if all measurements are done using the same conditions then they could be compared on the same body without any problems/issues . For comparison it important that all done exactly the same with the same conditions
If I want to compare lenses on the same body this is all OK.
As for luminance conditions - this could be considered as some negative shift/bias that could be taken into account if needed.
For me I use DXO data to see what could be the best performance lens for particular body so I do not have any problems with their measurements.

 

[Larsskv]

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.

I personally do not see any issues with the results of their measurements (not scores) looking at their data and doing my own tests. One interesting thing though that using best canon glass on my Sony a7rII I get perceptially better results than DXO publish for Canon 5DSR.

Consider what you are writing here. You do not see any issues, but you get better "perceptually better results" with the A7RII than the 5DSR. Considering that extra resolution, that seems a little weird, don´t you think?

Did you compare apples to apples?

Did you take your test picture in a room with 150 lux, at 1/60 sec, at base ISO and pushed the file to correct lighting when you took your comparison picture? No? Because that is what DXO seems to do. I get "perceptually better results" with all my f/4 lenses, than what is reported by DXO. Why? Because I make a correct exposure in a sufficient amount of light, when I am shooting, and therefore DXO is irrelevant for my type of shooting, which I guess is quite common - at least with Canon shooters.. DXO is very misleading, and it´s hard to know when they don´t let you know how they rate their scores.

(Edit: I don´t see why my text is quoted here...)

[

I agree that using this combination of ISO, shuter speed and luminance creates a bit difficult overall setup and gives latest better tech sensors some advantage.
With higher luminance results could be better.
And this applies in particular to the a7rII sensor. As I mentioned I can push sharpness very agressively using 0.5 radius without seeing any side effects. As results on final processed image I can see that my EF24-70 f/2.8L IS USM II resolve to every single pixel on a7rII. The same for the latest 70-200 and 100-400 .
So better visual results is contribution of both the lens quality and sensor quality and indicator of this combo performace.

On the other hand if all measurements are done using the same conditions then they could be compared on the same body without any problems/issues . For comparison it important that all done exactly the same with the same conditions
If I want to compare lenses on the same body this is all OK.
As for luminance conditions - this could be considered as some negative shift/bias that could be taken into account if needed.
For me I use DXO data to see what could be the best performance lens for particular body so I do not have any problems with their measurements.

DXO does not tell you how the lens performs at it´s sharpest aperture, given you have sufficient light. Further, one must be aware that wide aperture lenses have a major advantage over smaller aperture lenses, such as f/4 lenses. That is because of the peculiar lighting set up, 150 lux, and 1/60 sec shutter speed. Since I never shoot lets say f/4 lenses under such conditions, I find their measurements irrelevant.

 

[Aglet]

Perhaps you don't have any problem with unethical behavior..

oh, the irony

DxO Image Science...Bad Science...Biased Scores...BS.

doesn't bad ol' Dr. Brain use DXO Pro software on his little mac computer at home?

 

[3kramd5]

Perhaps you don't have any problem with unethical behavior..

oh, the irony

DxO Image Science...Bad Science...Biased Scores...BS.

doesn't bad ol' Dr. Brain use DXO Pro software on his little mac computer at home?

Pardon my skepticism that those people who turn lens data into useful image processing code are the same people turning lens data into a web-accessible database and scoring system.

The composite scores are bathwater.

 

[Mt Spokane Photography]

I personally do not see any issues with the results of their measurements (not scores) looking at their data

Sometimes yes, there could be errors in measurements or records due to number of reasons.
Most outstanding case was with Canon EF 70-200 f/2.8L IS USM II when DXO reported lower perceptial resolution compared to the previous generation, did not publicully admitted that this was wrong but eventially corrected measurements results.

Perhaps you don't have any problem with unethical behavior, but when an organization purports to be scientifically-based but fails to fully disclose their methods, they are not acting in accordance with the established practice of science. Moreover, when they are made aware of an error and defend their data rather than publicly acknowledging their error, then subsequently alter their underlying data but fail to correct their conclusions based on those faulty data, that behavior is counter to the ethical principles to which scientists are expected to adhere.

DxO Image Science...Bad Science...Biased Scores...BS.

Yes, I ran a Aerospace Lab, and would have been fired if I produced reports for NASA or other organizations that were so sloppy.

First, I had to use recognized and documented test methods and they had to make sense in the context that they were used for. I had to document the test setup and record all the results. The original data was always available for review. I was not allowed to make up my own tests.

Unfortunately, the lack of standard test methods applies to all camera testers, there is no standard test method for lenses attached to digital cameras, so the results we see can not be taken as reliable.

There are some popular tests that use software called Imatest, but it is not a accepted standard, and results definitely have flaws because they can not account for so many variables in a lens/camera combination. Even so, Imatest results can be duplicated by anyone willing to take the time and purchase the software.

Numbers invented by DXO like mpix are undefined and their use in final test scores is very confusing. A product can do very well in all the published results and get a lower score than a product that received lower results. Its because their methods don't make sense as far as pushing out a overall score.

 

[Don Haines]

Perhaps you don't have any problem with unethical behavior..

oh, the irony

DxO Image Science...Bad Science...Biased Scores...BS.

doesn't bad ol' Dr. Brain use DXO Pro software on his little mac computer at home?

I think there is a huge difference between the two......

The software has a well defined goal and allows the user to control it to their desired preferences.

The camera and lens rating is pre-determined mush where all the variables are pre-defined to values that are realistic.

If the software was similar, it would permananently set the white balance slider way over to one side, tint to the other side, boost gain by 5 stops, and only allow highly compressed jpg output at 640x400 resolution.

 

[Don Haines]

The problem with DXO's single number rating system is a conceptual problem. We can argue over how that number is determined, but in the end, it does not matter. The problem is the concept.

Any attempt to reduce a complex system used by diverse people for diverse goals down to a single rating metric is doomed to failure.

 

[candc]

I will be captain obvious. They compile a lot of specific data and then diminish its usefulness by combining it all into one number.

For instance. The sharpness value is an average across the field at all apertures. Maybe that's important to know for some reason? I don't care how sharp it is stopped all the way down, I don't use lenses that way.

 

[Kwwund]

Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?

There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are doomed.

 

[Loswr]

Well...consider the service provided by a restaurant critic. He tells you that the food is good, the service was slow, the ambience was poor, and the price was reasonable. Two out of four stars. You don't know the technical specs of the ratings on any dimension, nor the weights applied to each dimension. Is the 2-star rating useless because you don't know how it was mathematically determined?

There are lots of composite rating systems in all fields. They do tend to obscure detail, and they force the reader to trust the values/judgment/biases of the reviewer. But I would not go so far as to say that such composite ratings are doomed.

It the restaurant critic's motto, Food Science? :