canon rumors FORUM

Gear Talk => EOS Bodies - For Stills => Topic started by: aj1575 on September 02, 2013, 04:19:58 PM

Title: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 02, 2013, 04:19:58 PM
Hello, I made a table with different samples from different cameras at different settings. I took the pictures from dpReview, and arranged them in 6 rows with 4 pictures each. The picture in each row are taken with the same setting (ISO, Raw or JPG). The pictures are mainly from Canon or Nikon cameras, but others are also included.

Feel free to rank the pictures in each row, or to just point a interesting things. I will post the picture with the settings and the cameras later.

You should download the picture to see it at 100%

The goal is not to spot the 70D. The idea is to rank the picture according to your impression, and then see later from what cameras they were.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 02, 2013, 04:47:52 PM
Ok, I'll kick: :)
3A is the Canon 7D
3B is the Nikon D7100
3C is the Sony A77
3D is the Canon 70D
Did I kicked too far? ???
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 02, 2013, 06:38:51 PM
Someone else will guess?
Title: Re: IQ comparison; or how meaningful is DXO
Post by: JohnUSA on September 02, 2013, 07:56:33 PM
Here's my picks:

1. D
2. D
3. D
4. A
5. A
6. A
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 03, 2013, 01:32:24 AM
Ok, I'll kick: :)
3A is the Canon 7D
3B is the Nikon D7100
3C is the Sony A77
3D is the Canon 70D
Did I kicked too far? ???

Mostly Wrong. Try to sort the rows from best to worst, I will post the pic the names on it soon.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 03, 2013, 02:03:12 AM
I will do a ranking.

1. DCBA
2. DACB
3. DBAC
4. BDCA
5. DCAB
6. DABC
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Kristofgss on September 03, 2013, 03:31:40 AM
Do you mean sort per row or sort once to pick the most pleasing one of the four cameras?
Title: Re: IQ comparison; or how meaningful is DXO
Post by: luigig75 on September 03, 2013, 03:44:00 AM
1) DCBA
2) DCAB
3) DBAC
4) DBCA
5) ABCD
6) DABC
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 03, 2013, 04:12:15 AM
Do you mean sort per row or sort once to pick the most pleasing one of the four cameras?

You can also only pick the best and the worst. What I like to do is a blind test; just look at the pictures, without knowing what camera it is, and what the settings were.

It is very interesting. I made the table, but I can remember the position of only a few cameras, so there is nothing else to do than just look carefully at the pitures, and rate them by what I see.

I think I will do a table with only the DXO mark attached to it, this would also be interesting.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Cannon Man on September 03, 2013, 04:21:43 AM
I spot that they are all bad.. None of them a 1DX
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Kristofgss on September 03, 2013, 04:23:31 AM
1. acbd
2. cabd
3. dbac
4. bcda
5. dcab
6. adbc
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Woody on September 03, 2013, 04:51:31 AM
My ranking:

1 DCBA
2 DACB
3 DBAC
4 BDCA
5 BACD
6 DABC
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 03, 2013, 09:44:08 AM
I like to see some more ratings, so that it makes sense to put the results into some nice graphs.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: rpt on September 03, 2013, 09:56:12 AM
1. DCAB
2. ADCB
3. DABC
4. BDAC
5. ACBD
6. ADBC

There you go...
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 03, 2013, 11:58:16 AM
I'll try again. From best to worst ...
1-  D- C- A- B-
2-  D- A- B- C-
3-  D- B- A- C-
4-  B- D- A- C-
5-  C- A- B- D-
6-  D- A- B- C-
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Mt Spokane Photography on September 03, 2013, 12:07:48 PM
DXO measures the sensor characteristics.  Those are hard facts.  DXO does not test cameras.
 
For those that understand this, the information can be useful in selecting a camera that has a sensor which meets your special needs.
 
Unless its a extreme application, the photographer is 90%, the lens is 9.99 % and the body makes little difference to IQ.
 
Since DXO does not test bodies for other things that may matter to photographers like metering accuracy, their attempting to put a number to a camera body based on sensor testing is of little value to me, but the raw data is valuable.
 
However, if you are one who needs a particular sensor characteristic, understanding the strengths and weaknesses of your sensor might be important.  So far, I have found lenses to be so much more critical, that the sensor makes little or no difference, unless its defective.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Joynt Inspirations on September 03, 2013, 12:10:57 PM
I'm game.

1. DCBA
2. ADCB
3. DBAC
4. BDCA
5. ACBD
6. DBAC

Fool me once …
Title: Re: IQ comparison; or how meaningful is DXO
Post by: hutjeflut on September 03, 2013, 01:18:30 PM
its just not possible to siply rank them as each row has different pixtures that have different qualitys better like colour sharpness and noise are not always equally good on each shot.

1.  D simply beats all the others both in detail and sharpness but on it seems somewhat lower on the exposure side as its a little darker resulting in lower saturation.

2. D and A seem the have about the same iso preformance but A seems to be slightly out of focus and a little flat colour wise while C seems to be best colour wise but had poor details and noise preformance.

3. D is the best on everything here

4. B seems to have the best noise preformance but seeing all the details of the blocks has fallen away this immage on full size will look flat and lack detail so your mostlikely better off with D which had the second best noise preformance but maintains detail.

5. D has the best noise preformance but lacks detail and saturation and contrast making me think C would give the best end result  as its luminous noise mostly and has plenty detail and saturation. A has more detail but far worse noise preformance making you lose detail on noise reduction.

6. B has the best colour reproduction and a nice even move from dark to light blue while A as the most detail/sharpness/contrast and D has the best noise preformance.

so depending on what you look for different photos might fit your need best.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 04, 2013, 02:31:19 AM
its just not possible to siply rank them as each row has different pixtures that have different qualitys better like colour sharpness and noise are not always equally good on each shot.

....

so depending on what you look for different photos might fit your need best.

Thanks, a very nice analysis.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 04, 2013, 02:52:36 PM
Here is the comparison with the camera models and the settings of the pictures taken.

I leave it to you to judge your results; I will make some kind of statistics later.

Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 04, 2013, 03:03:11 PM
Here is the comparison with the camera models and the settings of the pictures taken.

I leave it to you to judge your results; I will make some kind of statistics later.
Well, I guessed right the line 3-D image was even canon 70D. :D Overall, my preferences this blind test, was also Canon 6D and 70D. By coincidence, are these cameras I would buy right now. ;)
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 04, 2013, 04:03:49 PM
It did not seem very fair to compare images of APS-C vs full frame. But in the end, the results were what one might expect. No big surprise. Just a little disappointed with the sharpness of the D7100, which having no AA filter, I expected to be sharper than other APS-C. ::)
Title: Re: IQ comparison; or how meaningful is DXO
Post by: privatebydesign on September 04, 2013, 08:42:59 PM
I came out with this, I didn't cheat for what that is worth.

D,C,B,A
A,D,C,B
B,C,D,A
B,C,D,A
C,D,A,B
D,A,B,C

So the 70D didn't fair too well for me, but the 6D did. However small crops of jpegs that haven't been normalised really don't illustrate much.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 04, 2013, 10:12:47 PM
What does this have to do with DXO?
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 04, 2013, 10:28:11 PM
What does this have to do with DXO?
In this blind test, each person chose the images that seemed best. Now we know what we were looking cameras, can compare with the DXO ranking. In my case, the images that I found most enjoyable were 70D and 6D. Two models with DXO scores worse than the competitors. For me, no surprise. The criteria DXO does not show which camera is the ideal image to my eyes.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: privatebydesign on September 04, 2013, 10:38:15 PM
It depends on what you think the images illustrate.

I don't consider them relevant to my buying decisions for two key reasons:
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 04, 2013, 10:42:10 PM
What does this have to do with DXO?
In this blind test, each person chose the images that seemed best. Now we know what we were looking cameras, can compare with the DXO ranking. In my case, the images that I found most enjoyable were 70D and 6D. Two models with DXO scores worse than the competitors. For me, no surprise. The criteria DXO does not show which camera is the ideal image to my eyes.

Some of the crops are OOF, you have different NR applied to them, etc. Your title is too provocative for what you are doing. You do not need to take cheap shots at DXO.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: CR00 on September 05, 2013, 02:55:38 PM
I think he mentioned at the beginning of the post that he took the pictures from dpReview, and arranged them in 6 rows with 4 pictures each.  I think you can try it at http://www.dpreview.com/reviews/nikon-d7100/20 (http://www.dpreview.com/reviews/nikon-d7100/20) and see it for yourself.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: dtaylor on September 05, 2013, 04:12:39 PM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: qwerty on September 05, 2013, 05:25:38 PM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 05, 2013, 05:34:16 PM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 05, 2013, 06:37:26 PM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: dtaylor on September 05, 2013, 09:00:50 PM
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).

Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Robert Welch on September 05, 2013, 11:22:26 PM
When it comes to applying the DXO ISO ratings to actual usage, I once heard someone suggest to round their ISO rating up to the next standard ISO setting (i.e. 1000 to 1600, 2000 to 3200), then double that. The result will generally be a reasonable high ISO to use the camera on with decent results for many applications.

In my experience, this translation has applied pretty well for the cameras I've owned, to some degree. My 40D & 7D are DXO rated about 700, and I generally don't like to use them much above 1600 if I can help it (the 7D does a little better if pushed higher, but some of that may be because of the higher resolution, meaning enlargement isn't as high). My 1DmkIII is rated about 1000, and while I feel at 3200 it's being pushed a bit, it is usable there, but I try to keep it no more than 2000-2500 if I can. The 6D & 5Dmk3 are both rated just above 2000, and both those cameras work very well at 6400. So, in broad strokes, this method seems to have some validity.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: qwerty on September 06, 2013, 12:14:23 AM
I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).

Yes there is. The biggest debate I've seen in other forums is over their DR scores which do not match the results from other testers (dpreview; IR) and do not seem to match real world experience. IMHO a simple Stouffer transmission step wedge test is far more accurate and reliable then DxO's methodology.

DxO also "interprets" DR based on output/viewing size, which is absurd to me.

Sorry, too long of a day for a fully coherent post, but:

If I am not mistaken, DPreview reports the dynamic range of processed jpegs, not of the raw files themselves.  Basing the measurement on processed files means that the DR reported depends on the processing applied during conversion (which is why they have several different dynamic ranges reported for each camera at each ISO).  (Also, the files they post for "raw" comparisons are obviously not really the raw files; I believe they are ACR conversions to jpeg from the raw files). 

DxO's real business is making raw converters; they analyze the sensor outputs to optimizer their converters (which is why they look at the actual raw data, not processed images).  They would tell you that the DxOmarks are just a side effect of their core business...

With regard to DxO's normalization, I think it would be absurd to compare non-normalized results.  Normalization tells you how things will perform for a fixed print size (which is what most of us care about, as opposed to per-pixel values).

I checked their math a few years ago and they do it right (see http://www.dxomark.com/index.php/Reviews/Detailed-computation-of-DxOMark-Sensor-normalization  (http://www.dxomark.com/index.php/Reviews/Detailed-computation-of-DxOMark-Sensor-normalization) and check for yourself if you do not believe me).

Comparing non-resolution-normalized results for cameras with different sizes is akin of comparing prints of different sizes.  It would be like comparing a 4x5 print from a 4 MP 1D with 8x10 from a 18 MP 1DX (from the same viewing distance)...


... ... ...
But, all that aside, if you just click the "screen" button when viewing the DxO results, they will happily give you the non-normalized values you want.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: jebrady03 on September 06, 2013, 12:19:54 AM
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 06, 2013, 12:33:00 AM
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better!  ;)
Title: Re: IQ comparison; or how meaningful is DXO
Post by: jebrady03 on September 06, 2013, 12:49:45 AM

This resolves a long standing question. Canon is better!  ;)

AGREE WHOLEHEARTEDLY!  lol ;)

But seriously, I've always said, and will continue to say, that the photographer > lenses > camera and if a person is iffy about which camera to buy, hold them and play with them.  Very often, ergonomics will decide.  After all, nobody wants to lug around a camera they don't enjoy using.

...and that's how I'm going to let them down easy ;)  lol

Any of the cameras in that test are GREAT choices and anyone who owns them should be VERY satisfied with their choice.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: ajfotofilmagem on September 06, 2013, 12:55:01 AM
I shared the first image on facebook and tagged a bunch of my friends and asked them to pick their favorites.  I haven't posted the results yet but so far, the preference for Canon is OVERWHELMING.  The greatest part about it is that the majority of the people I tagged are DIEHARD Nikon shooters who trash Canon every chance they get, and they're picking Canon left and right.  Only one guy split Canon and Nikon down the middle at 3 and 3 (and he's one of the more vocal Canon-haters) and everyone else is at least 2-4 with some being 1-4 (Nikon:Canon).
Crow... it's what's for dinner!

This resolves a long standing question. Canon is better!  ;)
The question is not as simple as: "Canon is the best!" The real question is: Pictures of Canon cameras are more pleasing to the eye when made and processed on equal terms, for most uses in the real world. ;) However, there are exceptions. ??? If I need to take pictures in RAW only, exclusively at ISO 100, with only prime lenses stopped down 3 stops from maximum aperture diaphragm, only objects with great brightness variation, certainly has the best Nikon camera for that. ::) For all other types of photo, the Canon system seems more advantageous, with images closer than I think it should look like. Obviously some people disagree, but the sun rises for all, and would not benefit a monopoly Canon, or any other company. 8)
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Woody on September 06, 2013, 02:17:48 AM
After comparing my image choices vs actual cameras used, this is what I notice:
i) FF is much better than cropped cameras
ii) Canon jpeg engine rules
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Apop on September 06, 2013, 02:40:43 AM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

I do not think there is much disagreement that their sensor tests, methods, and results are accurate and meaningful (at least for some uses).  The disagreements are to 1) whether the reported scores (scores, as opposed to test results) are fair, useful, meaningful or what-have-you 2) whether the differences matter for a given user and 3) the fact the DxO only measures sensor performance, not camera performance (and does not claim to do anything different).
That is why it is said: "Statistics is the prostitute of mathematics". ??? And also: "Statistics is a form of lying, using numbers". :-X What is the use of a collection of correct data, if the end result will be totally subjective score, and mysterious criteria? :-\

This is left to the intelligence of the reader. The "end result" has nothing to do with statistics, it is some kind of cumulative score for readers who are too busy to try to understand the data. The data is there for everybody who cares; the score is not data and every intelligent user would ignore it.

yes ,but
The scores/numbers should at least represent the data ?
Regardless of ones intelligence , some people are not interested or don't have the time to interpret the data themselves. I think DXO should deserve credit for conducting all these tests, if you agree or disagree with their scores/data shouldn't matter. They are putting in a lot of time and effort and if you do disagree , set up an experiment and try to disprove them rather than bashing them because your camera doesn't score the highest:P( this is just a general opinion of me, not aimed at you at all pi)

Still I think the score is derived from data and should not necessarily be ignored, since the score should represent (some of) the data.

For example; does anyone know where the image quality of a sensor is based on @ dxo?
95 vs 82 image quality, sensor 1 has around 20% better image quality.
To be honest I never value that score, because I have no clue where it is based on
(Resolution? DR? Noise? , no clue).

In that aspect I agree that scores should be ignored , they might have made up some formula with several variables and come up with such a score. again I have no clue how they get the number.

On the other hand , thins like
2.853 ISO     or    2.340 ISO     don't have to be ignored, If it's shot RAW and was a good test ( so repeated by others with the same result), that is enough information for me.



 
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 06, 2013, 03:22:46 AM
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Apop on September 06, 2013, 04:52:29 AM
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.



It's a nice thing you showed, but does DP review even use the same lenses for the cameras?( for example all sigma 35 1.4?, same apertures?, don't think so).
(One  thing i noticed they have trouble framing every camera in exactly the same way , and there are also quite some focus differences , making it a bit hard to compare.)

But in general I guess it's safe to say they are  close.(70d/d7100).

I think to say that DXO is complete nonsense isn't fair.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: AmbientLight on September 06, 2013, 05:29:41 AM
I think to say that DXO is complete nonsense isn't fair.

I disagree on this point. How can criticism be unfair, if DXO's ratings can at best be said to provide a distorted view?

We have to assume that DXO has some form of interest of misinterpretations. Otherwise we must decry them as unprofessional, which I assume they are not, so there must be some sort of intend behind those misleading ratings. To claim that all is good based on underlying correct measurements is not sufficient nor is it good enough. For example half-truths may not be outright lies, but they are not to be trusted either. If we must strain ourselves to properly interpret DXOs findings then there are at least some obvious shortcomings. In light of DXO rating's discrepancies with what others call reality or if this is not a good enough description in light of their rather mysterious way of creating a rating, calling their ratings complete nonsense is a sufficient first approximation of what they do.

If this were not so, you would have to make a claim that their ratings are useful as they are.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 06, 2013, 06:02:33 AM
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Apop on September 06, 2013, 07:06:21 AM
What I wanted to show is, that the DXOmark score is nonsense. Their measurements are nice, but even they lack information to conclude which camera makes the best pictures. Sure, there are some things that can be derived from the DXO numbers, but others not. One is for example noise; you can have the same amount of noise for two cameras, but to the human eye they look different, because of the patterns and the colors they appear in.

I also found the test interessting, because it even worked for myself, since I forgot most positions of the cameras, and also did a blind test (and I judged the pictures differently then when I knew from what camera they were).

My conclusion.
-The DXOmark score difference between the 70D and the D7100 is definitly not justified.
-The Fujifilm x-pro1 makes some nice pictures.
-The Sony a99 is a bit dissapionting, the D7100 and the 70D produce pictures that are about on the same level.
-FF is better, but not but the difference is not as big as I thought (the 70D was often rated higher than the D600 at JPEG).

I really tried to make a fair test; I took samples from colorcards to show noise performance at low ISO, I took parts with high contrast and some with details. So I think the comparison is quite fair. If it is meaningful to you, I don't know, this is up to you.

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Nokia :p?

If people want to proof that DXO is meanigless they need to do a bit better than posting samples like this.
See if you can replicate their results , and hope that they were wrong , then people try replicate yours and hope you were right ! :p
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 06, 2013, 07:29:31 AM
I think to say that DXO is complete nonsense isn't fair.

I did not say that DXO is complete nonsense, I said that the DXOmark score is nonesense.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: aj1575 on September 06, 2013, 07:47:48 AM

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Two things here, dp uses different lenses. For Canon APS-C they use a EF 50mm f1.4 for Nikon the 50mm f1.4 AF-S and for the FF they use the 85mm f1.8 from each brand. They shot at f8. So the differencr in quality should be rather small (check the tests of these lenses at various sites, also DXOmark)

The OOF claim is understandable. I was thinking the same when I looked at the samples. The D600 looks worse in the poker card than the 70D. But I do not think that this is an OOF problem, for several reasons.
1. The 7100D and other Nikons show this problem, so either they focus many Nikon cameras wrong, but not the Canons; or the Nikon AF is not accurate enough...
2. This softnes does not appear on the whole picture, you just need to move down on dp-site tool a little bit to the black and white circle. This seems to be on the same plane, but it looks sharp enough to me.

So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: Pi on September 06, 2013, 08:13:25 AM

You do not really think that posting selected OOF crops of JPEGs with different noise reduction proves anything, do you?

Tell me which brand you want to see a winner, and I will post similar crops from IR proving that that brand is the best.

Two things here, dp uses different lenses. For Canon APS-C they use a EF 50mm f1.4 for Nikon the 50mm f1.4 AF-S and for the FF they use the 85mm f1.8 from each brand. They shot at f8. So the differencr in quality should be rather small (check the tests of these lenses at various sites, also DXOmark)

How about DOF and focus plane differences?

Quote
So to me, this is not an OOF problem, the Nikon sensors just have a problem to resolve that properly.

Tell me that you are kidding.

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=640&Camera=614&Sample=0&FLI=0&API=2&LensComp=287&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=2 (http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=640&Camera=614&Sample=0&FLI=0&API=2&LensComp=287&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=2)

D7100:
(http://4.static.img-dpreview.com/reviews/nikon-d7100/samples/comparison/d7100DSC_0193.acr-crop2.jpg?v=2471)

Title: Re: IQ comparison; or how meaningful is DXO
Post by: Mt Spokane Photography on September 06, 2013, 10:56:26 AM
DXO measures the sensor characteristics.  Those are hard facts. 

No, they are the results of DxO's testing process and interpretations. There's quite a bit of disagreement as to whether or not DxO's tests are accurate and/or meaningful.

As someone who ran a lab for NASA, I think the test data  represents hard facts.  I've been thru hell repeating and defending my tests by experts who are skeptics, so I can recognize when someone is incompetent. I also think they are accurate results for the samples they had.   someone can always find a unit that performs differently.   You should be able to create the same setup and compare results.
 
Now, interpretation of the data is where people tend to disagree, and assigning a value number to a sensor based on a secret formula is silly, particularly when you link it to a camera model and give the camera a score.  There is a lot more to a camera than the sensor, and DXO does not test that.
Title: Re: IQ comparison; or how meaningful is DXO
Post by: qwerty on September 06, 2013, 03:47:59 PM
Just to try to drag the thread back on-topic, here are my favorites in each row:
1d
2a
3d
4b
5c (d had less noise, but seemed oversmoothed)
6d (d had less noise and seemed "sharp enough", but I get the feeling that if there were more detail it would have looked oversmoothed)

I want to thank the OP for making this comparison, and would be curious to see more of the same.  I know a few issues have been pointed out in the thread, and a second pass might be even better.

If I had the time, what I would do is download the raw files of a standard scene from somewhere (I thought dpreview posted them somewhere, but either I am misremembering, bad at searching, or they took them down when they added their scene comparison widget) and convert them yourself using lightroom with the same settings, except scaled so that the image sizes are the same for the different cameras.  You can even keep them as tiffs until final conversion to keep everyone (less un-)happy.

Oh, and how about a less inflammatory title for the next thread? : o )