IQ comparison; or how meaningful is DXO

Status
Not open for further replies.
aj1575 said:
Here is the comparison with the camera models and the settings of the pictures taken.

I leave it to you to judge your results; I will make some kind of statistics later.
Well, I guessed right the line 3-D image was even canon 70D. :D Overall, my preferences this blind test, was also Canon 6D and 70D. By coincidence, are these cameras I would buy right now. ;)
 
Upvote 0
I came out with this, I didn't cheat for what that is worth.

D,C,B,A
A,D,C,B
B,C,D,A
B,C,D,A
C,D,A,B
D,A,B,C

So the 70D didn't fair too well for me, but the 6D did. However small crops of jpegs that haven't been normalised really don't illustrate much.
 
Upvote 0
Pi said:
What does this have to do with DXO?
In this blind test, each person chose the images that seemed best. Now we know what we were looking cameras, can compare with the DXO ranking. In my case, the images that I found most enjoyable were 70D and 6D. Two models with DXO scores worse than the competitors. For me, no surprise. The criteria DXO does not show which camera is the ideal image to my eyes.
 
Upvote 0
It depends on what you think the images illustrate.

I don't consider them relevant to my buying decisions for two key reasons:
[list type=decimal]
[*]They are not normalised, that is, they are different sizes at 100%, this is not how I use images, I want a comparison of a set reproduction size, if I am making a 12" x 18" print I don't care how many pixels I have, I just need that size.
[*]I don't have any use for jpegs, indeed I practically don't use them at all, I even send 16 bit data to my printers now. Jpeg output is irrelevant to me, I need to know what I can get out of the RAW file and how much time it takes to get it.
[/list]
 
Upvote 0
ajfotofilmagem said:
Pi said:
What does this have to do with DXO?
In this blind test, each person chose the images that seemed best. Now we know what we were looking cameras, can compare with the DXO ranking. In my case, the images that I found most enjoyable were 70D and 6D. Two models with DXO scores worse than the competitors. For me, no surprise. The criteria DXO does not show which camera is the ideal image to my eyes.

Some of the crops are OOF, you have different NR applied to them, etc. Your title is too provocative for what you are doing. You do not need to take cheap shots at DXO.
 
Upvote 0
dtaylor said:
Mt Spokane Photography said:
DXO measures the sensor characteristics. Those are hard facts.

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses). The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
 
Upvote 0
qwerty said:
dtaylor said:
Mt Spokane Photography said:
DXO measures the sensor characteristics. Those are hard facts.

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses). The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\
 
Upvote 0
ajfotofilmagem said:
qwerty said:
dtaylor said:
Mt Spokane Photography said:
DXO measures the sensor characteristics. Those are hard facts.

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses). The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.
 
Upvote 0
qwerty said:
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).

Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.
 
Upvote 0
When it comes to applying the DXO ISO ratings to actual usage, I once heard someone suggest to round their ISO rating up to the next standard ISO setting (i.e. 1000 to 1600, 2000 to 3200), then double that. The result will generally be a reasonable high ISO to use the camera on with decent results for many applications.

In my experience, this translation has applied pretty well for the cameras I've owned, to some degree. My 40D & 7D are DXO rated about 700, and I generally don't like to use them much above 1600 if I can help it (the 7D does a little better if pushed higher, but some of that may be because of the higher resolution, meaning enlargement isn't as high). My 1DmkIII is rated about 1000, and while I feel at 3200 it's being pushed a bit, it is usable there, but I try to keep it no more than 2000-2500 if I can. The 6D & 5Dmk3 are both rated just above 2000, and both those cameras work very well at 6400. So, in broad strokes, this method seems to have some validity.
 
Upvote 0
dtaylor said:
qwerty said:
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).

Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.

Sorry, too long of a day for a fully coherent post, but:

If I am not mistaken, DPreview reports the dynamic range of processed jpegs, not of the raw files themselves. Basing the measurement on processed files means that the DR reported depends on the processing applied during conversion (which is why they have several different dynamic ranges reported for each camera at each ISO). (Also, the files they post for "raw" comparisons are obviously not really the raw files; I believe they are ACR conversions to jpeg from the raw files).

DxO's real business is making raw converters; they analyze the sensor outputs to optimizer their converters (which is why they look at the actual raw data, not processed images). They would tell you that the DxOmarks are just a side effect of their core business...

With regard to DxO's normalization, I think it would be absurd to compare non-normalized results. Normalization tells you how things will perform for a fixed print size (which is what most of us care about, as opposed to per-pixel values).

I checked their math a few years ago and they do it right (see http://www.dxomark.com/index.php/Reviews/Detailed-computation-of-DxOMark-Sensor-normalization and check for yourself if you do not believe me).

Comparing non-resolution-normalized results for cameras with different sizes is akin of comparing prints of different sizes. It would be like comparing a 4x5 print from a 4 MP 1D with 8x10 from a 18 MP 1DX (from the same viewing distance)...


... ... ...
But, all that aside, if you just click the "screen" button when viewing the DxO results, they will happily give you the non-normalized values you want.
 
Upvote 0
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!
 
Upvote 0
jebrady03 said:
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better! ;)
 
Upvote 0
Pi said:
This resolves a long standing question. Canon is better! ;)

AGREE WHOLEHEARTEDLY! lol ;)

But seriously, I've always said, and will continue to say, that the photographer > lenses > camera and if a person is iffy about which camera to buy, hold them and play with them. Very often, ergonomics will decide. After all, nobody wants to lug around a camera they don't enjoy using.

...and that's how I'm going to let them down easy ;) lol

Any of the cameras in that test are GREAT choices and anyone who owns them should be VERY satisfied with their choice.
 
Upvote 0
Pi said:
jebrady03 said:
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites. I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING. The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right. Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better! ;)
The question is not as simple as: "Canon is the best!" The real question is: Pictures of Canon cameras are more pleasing to the eye when made and processed on equal terms, for most uses in the real world. ;) However, there are exceptions. ??? If I need to take pictures in RAW only, exclusively at ISO 100, with only prime lenses stopped down 3 stops from maximum aperture diaphragm, only objects with great brightness variation, certainly has the best Nikon camera for that. ::) For all other types of photo, the Canon system seems more advantageous, with images closer than I think it should look like. Obviously some people disagree, but the sun rises for all, and would not benefit a monopoly Canon, or any other company. 8)
 
Upvote 0
Status
Not open for further replies.