April 18, 2014, 05:07:11 AM

Author Topic: IQ comparison; or how meaningful is DXO  (Read 7538 times)

ajfotofilmagem

  • 5D Mark III
  • ******
  • Posts: 682
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #30 on: September 05, 2013, 05:34:16 PM »
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\
« Last Edit: September 05, 2013, 06:27:39 PM by ajfotofilmagem »

canon rumors FORUM

Re: IQ comparison; or how meaningful is DXO
« Reply #30 on: September 05, 2013, 05:34:16 PM »

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: IQ comparison; or how meaningful is DXO
« Reply #31 on: September 05, 2013, 06:37:26 PM »
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.

dtaylor

  • 7D
  • *****
  • Posts: 441
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #32 on: September 05, 2013, 09:00:50 PM »
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).

Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.

Robert Welch

  • Canon AE-1
  • ***
  • Posts: 78
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #33 on: September 05, 2013, 11:22:26 PM »
When it comes to applying the DXO ISO ratings to actual usage, I once heard someone suggest to round their ISO rating up to the next standard ISO setting (i.e. 1000 to 1600, 2000 to 3200), then double that. The result will generally be a reasonable high ISO to use the camera on with decent results for many applications.

In my experience, this translation has applied pretty well for the cameras I've owned, to some degree. My 40D & 7D are DXO rated about 700, and I generally don't like to use them much above 1600 if I can help it (the 7D does a little better if pushed higher, but some of that may be because of the higher resolution, meaning enlargement isn't as high). My 1DmkIII is rated about 1000, and while I feel at 3200 it's being pushed a bit, it is usable there, but I try to keep it no more than 2000-2500 if I can. The 6D & 5Dmk3 are both rated just above 2000, and both those cameras work very well at 6400. So, in broad strokes, this method seems to have some validity.
« Last Edit: September 05, 2013, 11:25:15 PM by Robert Welch »

qwerty

  • PowerShot G16
  • **
  • Posts: 49
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #34 on: September 06, 2013, 12:14:23 AM »
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).


Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.


Sorry, too long of a day for a fully coherent post, but:

If I am not mistaken, DPreview reports the dynamic range of processed jpegs, not of the raw files themselves.  Basing the measurement on processed files means that the DR reported depends on the processing applied during conversion (which is why they have several different dynamic ranges reported for each camera at each ISO).  (Also, the files they post for "raw" comparisons are obviously not really the raw files; I believe they are ACR conversions to jpeg from the raw files). 

DxO's real business is making raw converters; they analyze the sensor outputs to optimizer their converters (which is why they look at the actual raw data, not processed images).  They would tell you that the DxOmarks are just a side effect of their core business...

With regard to DxO's normalization, I think it would be absurd to compare non-normalized results.  Normalization tells you how things will perform for a fixed print size (which is what most of us care about, as opposed to per-pixel values).

I checked their math a few years ago and they do it right (see http://www.dxomark.com/index.php/Reviews/Detailed-computation-of-DxOMark-Sensor-normalization and check for yourself if you do not believe me).

Comparing non-resolution-normalized results for cameras with different sizes is akin of comparing prints of different sizes.  It would be like comparing a 4x5 print from a 4 MP 1D with 8x10 from a 18 MP 1DX (from the same viewing distance)...


... ... ...
But, all that aside, if you just click the "screen" button when viewing the DxO results, they will happily give you the non-normalized values you want.

jebrady03

  • EOS M2
  • ****
  • Posts: 205
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #35 on: September 06, 2013, 12:19:54 AM »
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: IQ comparison; or how meaningful is DXO
« Reply #36 on: September 06, 2013, 12:33:00 AM »
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better!  ;)

canon rumors FORUM

Re: IQ comparison; or how meaningful is DXO
« Reply #36 on: September 06, 2013, 12:33:00 AM »

jebrady03

  • EOS M2
  • ****
  • Posts: 205
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #37 on: September 06, 2013, 12:49:45 AM »

This resolves a long standing question. Canon is better!  ;)

AGREE WHOLEHEARTEDLY!  lol ;)

But seriously, I've always said, and will continue to say, that the photographer > lenses > camera and if a person is iffy about which camera to buy, hold them and play with them.  Very often, ergonomics will decide.  After all, nobody wants to lug around a camera they don't enjoy using.

...and that's how I'm going to let them down easy ;)  lol

Any of the cameras in that test are GREAT choices and anyone who owns them should be VERY satisfied with their choice.

ajfotofilmagem

  • 5D Mark III
  • ******
  • Posts: 682
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #38 on: September 06, 2013, 12:55:01 AM »
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better!  ;)
The question is not as simple as: "Canon is the best!" The real question is: Pictures of Canon cameras are more pleasing to the eye when made and processed on equal terms, for most uses in the real world. ;) However, there are exceptions. ??? If I need to take pictures in RAW only, exclusively at ISO 100, with only prime lenses stopped down 3 stops from maximum aperture diaphragm, only objects with great brightness variation, certainly has the best Nikon camera for that. ::) For all other types of photo, the Canon system seems more advantageous, with images closer than I think it should look like. Obviously some people disagree, but the sun rises for all, and would not benefit a monopoly Canon, or any other company. 8)
« Last Edit: September 06, 2013, 01:00:26 AM by ajfotofilmagem »

Woody

  • 6D
  • *****
  • Posts: 498
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #39 on: September 06, 2013, 02:17:48 AM »
After comparing my image choices vs actual cameras used, this is what I notice:
i) FF is much better than cropped cameras
ii) Canon jpeg engine rules

Apop

  • Rebel T5i
  • ****
  • Posts: 151
    • View Profile
    • Apophoto
Re: IQ comparison; or how meaningful is DXO
« Reply #40 on: September 06, 2013, 02:40:43 AM »
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.

yes ,but
The scores/numbers should at least represent the data ?
Regardless of ones intelligence , some people are not interested or don't have the time to interpret the data themselves. I think DXO should deserve credit for conducting all these tests, if you agree or disagree with their scores/data shouldn't matter. They are putting in a lot of time and effort and if you do disagree , set up an experiment and try to disprove them rather than bashing them because your camera doesn't score the highest:P( this is just a general opinion of me, not aimed at you at all pi)

Still I think the score is derived from data and should not necessarily be ignored, since the score should represent (some of) the data.

For example; does anyone know where the image quality of a sensor is based on @ dxo?
95 vs 82 image quality, sensor 1 has around 20% better image quality.
To be honest I never value that score, because I have no clue where it is based on
(Resolution? DR? Noise? , no clue).

In that aspect I agree that scores should be ignored , they might have made up some formula with several variables and come up with such a score. again I have no clue how they get the number.

On the other hand , thins like
2.853 ISO     or    2.340 ISO     don't have to be ignored, If it's shot RAW and was a good test ( so repeated by others with the same result), that is enough information for me.



 

aj1575

  • Rebel T5i
  • ****
  • Posts: 160
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #41 on: September 06, 2013, 03:22:46 AM »
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

Apop

  • Rebel T5i
  • ****
  • Posts: 151
    • View Profile
    • Apophoto
Re: IQ comparison; or how meaningful is DXO
« Reply #42 on: September 06, 2013, 04:52:29 AM »
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.



It's a nice thing you showed, but does DP review even use the same lenses for the cameras?( for example all sigma 35 1.4?, same apertures?, don't think so).
(One  thing i noticed they have trouble framing every camera in exactly the same way , and there are also quite some focus differences , making it a bit hard to compare.)

But in general I guess it's safe to say they are  close.(70d/d7100).

I think to say that DXO is complete nonsense isn't fair.
« Last Edit: September 06, 2013, 04:55:21 AM by Apop »

canon rumors FORUM

Re: IQ comparison; or how meaningful is DXO
« Reply #42 on: September 06, 2013, 04:52:29 AM »

AmbientLight

  • 6D
  • *****
  • Posts: 484
    • View Profile
Re: IQ comparison; or how meaningful is DXO
« Reply #43 on: September 06, 2013, 05:29:41 AM »
I think to say that DXO is complete nonsense isn't fair.

I disagree on this point. How can criticism be unfair, if DXO's ratings can at best be said to provide a distorted view?

We have to assume that DXO has some form of interest of misinterpretations. Otherwise we must decry them as unprofessional, which I assume they are not, so there must be some sort of intend behind those misleading ratings. To claim that all is good based on underlying correct measurements is not sufficient nor is it good enough. For example half-truths may not be outright lies, but they are not to be trusted either. If we must strain ourselves to properly interpret DXOs findings then there are at least some obvious shortcomings. In light of DXO rating's discrepancies with what others call reality or if this is not a good enough description in light of their rather mysterious way of creating a rating, calling their ratings complete nonsense is a sufficient first approximation of what they do.

If this were not so, you would have to make a claim that their ratings are useful as they are.

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: IQ comparison; or how meaningful is DXO
« Reply #44 on: September 06, 2013, 06:02:33 AM »
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

canon rumors FORUM

Re: IQ comparison; or how meaningful is DXO
« Reply #44 on: September 06, 2013, 06:02:33 AM »