IQ comparison; or how meaningful is DXO

[Apop]

DXO measures the sensor characteristics. Those are hard facts.

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses). The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).

That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.

yes ,but
The scores/numbers should at least represent the data ?
Regardless of ones intelligence , some people are not interested or don't have the time to interpret the data themselves. I think DXO should deserve credit for conducting all these tests, if you agree or disagree with their scores/data shouldn't matter. They are putting in a lot of time and effort and if you do disagree , set up an experiment and try to disprove them rather than bashing them because your camera doesn't score the highest( this is just a general opinion of me, not aimed at you at all pi)

Still I think the score is derived from data and should not necessarily be ignored, since the score should represent (some of) the data.

For example; does anyone know where the image quality of a sensor is based on @ dxo?
95 vs 82 image quality, sensor 1 has around 20% better image quality.
To be honest I never value that score, because I have no clue where it is based on
(Resolution? DR? Noise? , no clue).

In that aspect I agree that scores should be ignored , they might have made up some formula with several variables and come up with such a score. again I have no clue how they get the number.

On the other hand , thins like
2.853 ISO or 2.340 ISO don't have to be ignored, If it's shot RAW and was a good test ( so repeated by others with the same result), that is enough information for me.

 

[aj1575]

What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

 

[Apop]

What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

It's a nice thing you showed, but does DP review even use the same lenses for the cameras?( for example all sigma 35 1.4?, same apertures?, don't think so).
(One thing i noticed they have trouble framing every camera in exactly the same way , and there are also quite some focus differences , making it a bit hard to compare.)

But in general I guess it's safe to say they are close.(70d/d7100).

I think to say that DXO is complete nonsense isn't fair.

 

[AmbientLight]

I think to say that DXO is complete nonsense isn't fair.

I disagree on this point. How can criticism be unfair, if DXO's ratings can at best be said to provide a distorted view?

We have to assume that DXO has some form of interest of misinterpretations. Otherwise we must decry them as unprofessional, which I assume they are not, so there must be some sort of intend behind those misleading ratings. To claim that all is good based on underlying correct measurements is not sufficient nor is it good enough. For example half-truths may not be outright lies, but they are not to be trusted either. If we must strain ourselves to properly interpret DXOs findings then there are at least some obvious shortcomings. In light of DXO rating's discrepancies with what others call reality or if this is not a good enough description in light of their rather mysterious way of creating a rating, calling their ratings complete nonsense is a sufficient first approximation of what they do.

If this were not so, you would have to make a claim that their ratings are useful as they are.

 

[Pi]

What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

 

[Apop]

What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Nokia ?

If people want to proof that DXO is meanigless they need to do a bit better than posting samples like this.
See if you can replicate their results , and hope that they were wrong , then people try replicate yours and hope you were right !

 

[aj1575]

I think to say that DXO is complete nonsense isn't fair.

I did not say that DXO is complete nonsense, I said that the DXOmark score is nonesense.

 

[aj1575]

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Two things here, dp uses different lenses. For Canon APS-C they use a EF 50mm f1.4 for Nikon the 50mm f1.4 AF-S and for the FF they use the 85mm f1.8 from each brand. They shot at f8. So the differencr in quality should be rather small (check the tests of these lenses at various sites, also DXOmark)

The OOF claim is understandable. I was thinking the same when I looked at the samples. The D600 looks worse in the poker card than the 70D. But I do not think that this is an OOF problem, for several reasons.
1. The 7100D and other Nikons show this problem, so either they focus many Nikon cameras wrong, but not the Canons; or the Nikon AF is not accurate enough...
2. This softnes does not appear on the whole picture, you just need to move down on dp-site tool a little bit to the black and white circle. This seems to be on the same plane, but it looks sharp enough to me.

So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.

 

[Pi]

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Two things here, dp uses different lenses. For Canon APS-C they use a EF 50mm f1.4 for Nikon the 50mm f1.4 AF-S and for the FF they use the 85mm f1.8 from each brand. They shot at f8. So the differencr in quality should be rather small (check the tests of these lenses at various sites, also DXOmark)

How about DOF and focus plane differences?

So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.

Tell me that you are kidding.

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=640&Camera=614&Sample=0&FLI=0&API=2&LensComp=287&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=2

D7100:

 

[Mt Spokane Photography]

DXO measures the sensor characteristics. Those are hard facts.

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

As someone who ran a lab for NASA, I think the test data represents hard facts. I've been thru hell repeating and defending my tests by experts who are skeptics, so I can recognize when someone is incompetent. I also think they are accurate results for the samples they had. someone can always find a unit that performs differently. You should be able to create the same setup and compare results.

Now, interpretation of the data is where people tend to disagree, and assigning a value number to a sensor based on a secret formula is silly, particularly when you link it to a camera model and give the camera a score. There is a lot more to a camera than the sensor, and DXO does not test that.

 

[qwerty]

Just to try to drag the thread back on-topic, here are my favorites in each row:
1d
2a
3d
4b
5c (d had less noise, but seemed oversmoothed)
6d (d had less noise and seemed "sharp enough", but I get the feeling that if there were more detail it would have looked oversmoothed)

I want to thank the OP for making this comparison, and would be curious to see more of the same. I know a few issues have been pointed out in the thread, and a second pass might be even better.

If I had the time, what I would do is download the raw files of a standard scene from somewhere (I thought dpreview posted them somewhere, but either I am misremembering, bad at searching, or they took them down when they added their scene comparison widget) and convert them yourself using lightroom with the same settings, except scaled so that the image sizes are the same for the different cameras. You can even keep them as tiffs until final conversion to keep everyone (less un-)happy.

Oh, and how about a less inflammatory title for the next thread? : o )