DXO vs Reality

[infared]

I'm a Nikon shooter currently with a D5100. DxOMark ranks the D5100/D7000 very well in terms of DR, but I can never really get the amount of dynamic range that I'm happy with in my photos. This is primarily why I'm really looking to go full frame.

Just goes to show you how useless numbers are.

The DR ratings are like gas milage ratings. They are the best ideal case, but not what most users see. The DR ratings for jpeg are closer to reality. You can get more DR from Raw files, but the images more often than not look flat and awful.

Raw files are neutral..and FULL of information. That is the beauty of them. Yes they look flat. Take them to post and you have SO MUCH MORE information to pull out the image to create what you saw when you took the photo. Yes it takes more time, yes you need good software, (more than one in my opinion, depending on what the final image will look like)..but Raw Files are an incredible tool. They are the whole reason why I spend my hard earned money on a expensive camera that produces them. I know a guy who owns an 5D Mark II and uses it to shoot weddings in jpeg? Me thinks he might get a LITTLE more detail out of the bride's dress if he shot in Raw but my suggestions fell on deaf ears?

 

[elflord]

I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

You have to look beyond the one number summary and look at the graphs/scores. The medium format backs don't score as high because they don't good high ISO numbers. I don't know these products well, but it looks like they're not designed for high ISO shooting (Some simply won't let you push the ISO very high).

I suspect the real reason DxO is coming under fire all of a sudden has more to do with the Nikon D800's score than the scores of Medium format cameras.

 

[well_dunno]

I did mention a few times prior to D800 testing that DxO suggested Canon 70-200 f/2.8 IS mark 1 was sharper than mark 2. Even after forum complaints there that the results could not be correct, DxO maintained that there was nothing wrong with the results.

http://www.dxomark.com/index.php/News/DxOMark-news/Canon-EF-70-200mm-f-2.8L-IS-II-USM-measurements-and-review

This is probably the only source that suggests mark 1 was sharper than mark 2. I am not suggesting DxO is necessarily biased, but something does not seem right. Perhaps their testing just does not have enough coverage to give a realistic picture?

Either case, I look at their figures but never rely on them for any purchase...

Cheers!

 

[Maui5150]

I did mention a few times prior to D800 testing that DxO suggested Canon 70-200 f/2.8 IS mark 1 was sharper than mark 2. Even after forum complaints there that the results could not be correct, DxO maintained that there was nothing wrong with the results.

http://www.dxomark.com/index.php/News/DxOMark-news/Canon-EF-70-200mm-f-2.8L-IS-II-USM-measurements-and-review

This is probably the only source that suggests mark 1 was sharper than mark 2. I am not suggesting DxO is necessarily biased, but something does not seem right. Perhaps their testing just does not have enough coverage to give a realistic picture?

Either case, I look at their figures but never rely on them for any purchase...

Cheers!

A couple thoughts come to mind...

For me it was not so much the Nikon D800 score, but it was that score relative to other cameras I know to be better and more capable. If you go by DxO, the D4 and D3s are shite compared to the D800 which is just not true. Then when you throw the whole MF quality into the mix... Yes, the MFs may bot be designed for High ISO per se, but does beg to question the ability to quantitatively measure IQ

In someways reminds me of CDs versus LPs, and while "Digital" is supposed to capture and be so reproduceable, the audiophiles still find magic in the outdated LPs for sound quality.

Not to bring film versus digital back in the mix, but I do think that it is possible to produce a camera that "tests" well but whose IQ to the eye does not meet with the same judgement.

To me this diminishes the value of DxO since they are telling me a new BMW outperforms a new Ferrari

Is the D800 a better camera than the D4? DxO says it is by a big margin.

You can tell me Rosie O'donnell is hotter than Olivia Wilde, I'll say you just have a thing for fat chicks who aren't funny

 

[pj1974]

I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites. There are many shades of grey.

DxO has some very helpful tools and software. Just their sensor tests don't cut the mustard, imho.

Regards

Paul

 

[Maui5150]

...
For me it was not so much the Nikon D800 score, but it was that score relative to other cameras I know to be better and more capable. If you go by DxO, the D4 and D3s are S___e compared to the D800 which is just not true. Then when you throw the whole MF quality into the mix... Yes, the MFs may bot be designed for High ISO per se, but does beg to question the ability to quantitatively measure IQ

Then don't look at the score, look at the signal to noise ratio, colour depth, tonality and dynamic range graphs individually to see how the D800 scores relative to those other cameras. The thing is, DxO didn't give the D800 for one attribute alone, it was from many.

If you compare the DR of the D800 to any other DSLR, you'll find that at ISO under 400, and especially ISO 100, it has no competition.

To me this diminishes the value of DxO since they are telling me a new BMW outperforms a new Ferrari

That's not unheard of.

Just because a new Ferrari is new does not mean it is as good as or better than any other car.

Is the D800 a better camera than the D4? DxO says it is by a big margin.

DxO doesn't measure/score "camera", it scores "sensor." Thus DxO ranks camera based on sensor and sensor alone. For some people that will make it a better camera than the D4, for some it will not.

Your making my point. Sorry. The sensor in the d800 is not leaps and bounds over that of the D4.

I know that.
Nikon knows that.
DxO tests can't figure that out.

Despite what DxO is trying to tell me, Rosie O'Donnell is still fat and not more attractive than Olivia Wilde.

Fact. Not Opinion.

 

[rpt]

DxO doesn't measure/score "camera", it scores "sensor." Thus DxO ranks camera based on sensor and sensor alone. For some people that will make it a better camera than the D4, for some it will not.

Gold!

You hit the head one the nail (or the other way around... whatever... )

 

[itsnotmeyouknow]

I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

You have to look beyond the one number summary and look at the graphs/scores. The medium format backs don't score as high because they don't good high ISO numbers. I don't know these products well, but it looks like they're not designed for high ISO shooting (Some simply won't let you push the ISO very high).

I suspect the real reason DxO is coming under fire all of a sudden has more to do with the Nikon D800's score than the scores of Medium format cameras.

The only medium format digital camera with an ISO of above 800 is the Pentax 645D which goes up to 1600. Th ereason being that most, apart from teh Pentax are aimed at studio shooters. The Pentax, being weather shielded and having a few lenses that are also weather shielded is aimed also at landscape shooters, which is does very well.

This is a shot I took at a concert in Sweden which was at every extreme:

Pentax 645D A (manual focus) 80 - 160 f/4.5 @ f/4.5 and 160mm 1/125 ISO 1600


hannah with orchestra in sweden by singingsnapper, on Flickr

 

[marekjoz]

I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites. There are many shades of grey.

DxO has some very helpful tools and software. Just their sensor tests don't cut the mustard, imho.

Regards

Paul

Paul
since English is not my first language, sometimes it's more difficult for me to grade shades of grey and also sometimes it's more difficult to find what's thrown into one basket Anyway you clearly state what you like and dislike in DxO and this is fair. I just have problems with statements (it's not to you, Paul) I read: "they are wrong because the results are not what I expected, what I invested in, what I like, what I believe....".
It's obviously not logic because of this simple schema:
1. Did they describe their testing procedures? Yes
2. Do they follow their testing procedures during tests? Most probably, they risk too much.
3. Is there anything wrong with their testing procedures? If there is, then I'd like to read about that - why there is something wrong with the way they run their test.
4. Is there anything wrong with the interpretation of their results? Is yes, then I'd like to read about that.

I simply don't understand people, who fight against numbers, numbers having their interpretation. If someone believes there is so much wrong with DxO, then there is a simple way to proof it.
In my opinion: I do believe that numbers which DxO publishes are real. There might be sometimes something to interpretation but as far as I didn't see clearly pushing one brand over another and interpreting same or comparative results the opposite way depending on situation, then I'm not convinced they lie.

Paul, do you know what I mean?
Regards, Marek

 

[Actionpix]

I always am surprised how people can get excited over "theory". "Well diffraction is only theory but no real life." "Well calculating DoF/hyper focal focusing is no real life situation." It is. "Theory", a word "artists" hate, explains why your images get unnecessary soft when you use to much aperture. Theory explains why you do not get the max (technical) out of your images. The same one can say concerning lab tests. They are "no real life situation" but will show the potential of equipment, whether you use it in real life or not. The potential is there and it has values you can compare. If you do not like the result you can argue against "theory" and "lab test" but they are as real as real life. They only are emotionless.

 

[dichiaras]

Why are so many people so much against science and the scientific method?
DxO gives results for a standardized 8mp image as well as a 100% crop. The more important one is the former, since you usually crop just a little your image, and then downsize it. I looked also at the results for DR from dpreview (for 100% crop), and they more or less agree with those from DxO. So at the end of the day DxO and dpreview both agree that Nikon and Sony have better DR than Canon (crop sensors at least). Quoting from the dpreview of the 600D about DR: "Overall this is exactly what we're used to seeing from Canon DSLRs, meaning a little less highlight range by default than is typically obtained from Nikon and Sony competitors ".

DxO rates the image quality given by Canon lenses as higher than that of Nikon lenses: shall we believe that DxO gives unfair advantage to Canon for the lenses and to Nikon/Sony for the sensors? No, that's silly!

Since I'm at it, I think that dpreview is definitely more lenient with Canon than with any other brand: while describing the highlight priority option they write "Turn this on and the 600D captures an extra stop in the highlights, resulting in an overall range that at least matches that of Sony and Nikon models." That's inaccurate, since the Nikon D5100 has a similar option that pushes DR much higher than the 600D's, as written in the D5100 review. Between DxO and dpreview, I think it's the latter to be unfair.

But again, for many people Canon is a religion: they just have faith, and definitely not in numbers.

PS I have a Canon 1100D with four Canon lenses, and am very happy with the IQ that I get for the buck, but before upgrading (which I plan to do as soon as the 650D and 70D come out) I will take a long and accurate look at the DxO test results.

 

[marekjoz]

I always am surprised how people can get excited over "theory". "Well diffraction is only theory but no real life." "Well calculating DoF/hyper focal focusing is no real life situation." It is. "Theory", a word "artists" hate, explains why your images get unnecessary soft when you use to much aperture. Theory explains why you do not get the max (technical) out of your images. The same one can say concerning lab tests. They are "no real life situation" but will show the potential of equipment, whether you use it in real life or not. The potential is there and it has values you can compare. If you do not like the result you can argue against "theory" and "lab test" but they are as real as real life. They only are emotionless.

Sorry but it's Throwing tests away is like saying: "I don't care how many HP has my engine. I don't care how fast does it accelerate. Those are just numbers." o
Of course - those are just numbers comparing if one car is faster than the other. That's about numbers as people like to see things comparable. It doesn't say what comfort is inside, nor where and how gently you drive.
Another example - I don't care if my camera/lens resolves 1000lp/mm or 400 = I don't care if my camera has 8MP or 20MP = I don't care if my camera shoots 8fps or 3fps, 14 bits TR or 8 bits TR and so on...
Let's not get crazy. Not everything is comparable but there are important things that are. Better camera will not make a photographer of anybody - that's obvious. But why not look on comparisons, tests or numbers? I don't get it.
Charts and graphs are simple enhancement to specification. What's wrong in it?

 

[awinphoto]

For the most part, my view on DxO is similar to how i take DPR and the rest of the "in-depth" review sites... with a grain of salt. Getting a top of the line camera isn't going to make you a better photographer... It CAN however give you a higher ceiling to learn... It CAN give you more options and make things easier, faster, cleaner, but it will vary from photographer to photographer. Nikon, in theory, has typically scored higher than canon in the last half decade in these sort of tests, however a good chunk of National Geo's staff and field photographers shoot canon and get great images.

As far as DR and such, as another has posted, in post, it gives more flexibility and room for editing... while that's fine and good, your forgetting a lot of "pro" photographers, when it comes to photoshop or post production, are lazy. You would be surprised how many pro's outsource photoshop work. You would also be surprised how many dont outsource but "shoot to print"... They use an expodisc or the like, nail WB, nail exposure, shoot, and then do little in post to prep the files... If all of a sudden the files are turning up flat, that isn't going to sit well as it defeats their business motto. Of course there are some "low volume" photographers who charge more per shoot, they can afford spending time in PS or LR and clean up images all day long, but that is going to get old quick. So there's different angles to look at everything.

Lastly, coming from a background of shooting cameras that weren't the most opportunistic cameras, such as toyo's for LF instead of sinars, bronicas and kowas instead of Mayima and such... I have learned to adapt, to make the best of my equipment, and not to worry so much about numbers/test/reports as if i have, I would have dropped out a long time ago... In the end, it's not the equipment nor what some test says about the equipment, it's about how you use said equipment.

 

[awinphoto]

Why are so many people so much against science and the scientific method?
DxO gives results for a standardized 8mp image as well as a 100% crop. The more important one is the former, since you usually crop just a little your image, and then downsize it. I looked also at the results for DR from dpreview (for 100% crop), and they more or less agree with those from DxO. So at the end of the day DxO and dpreview both agree that Nikon and Sony have better DR than Canon (crop sensors at least). Quoting from the dpreview of the 600D about DR: "Overall this is exactly what we're used to seeing from Canon DSLRs, meaning a little less highlight range by default than is typically obtained from Nikon and Sony competitors ".

DxO rates the image quality given by Canon lenses as higher than that of Nikon lenses: shall we believe that DxO gives unfair advantage to Canon for the lenses and to Nikon/Sony for the sensors? No, that's silly!

Since I'm at it, I think that dpreview is definitely more lenient with Canon than with any other brand: while describing the highlight priority option they write "Turn this on and the 600D captures an extra stop in the highlights, resulting in an overall range that at least matches that of Sony and Nikon models." That's inaccurate, since the Nikon D5100 has a similar option that pushes DR much higher than the 600D's, as written in the D5100 review. Between DxO and dpreview, I think it's the latter to be unfair.

But again, for many people Canon is a religion: they just have faith, and definitely not in numbers.

PS I have a Canon 1100D with four Canon lenses, and am very happy with the IQ that I get for the buck, but before upgrading (which I plan to do as soon as the 650D and 70D come out) I will take a long and accurate look at the DxO test results.

It's not so much that we are against scientific tests, it's just not the all to be all... in this day and age, we are so competitive, so eager to be #1 that when something isn't #1, we get panicky... Case in point, D800 scores well in the DxO scores.... you saw how the forum threads blew up... oh what has canon done, nikon is so much better, DR this, DR that, when will canon catch up... shoot that basically summarized two full days on Canon Rumors last week. Even IF, and this is a big IF, canon comes even within 2-3 points on the ratings, how does that fair out when pairing Canons lenses and nikons lenses... All the obsessing over sensors is like the film days... which film has better DR, which film has the better color/saturation, which has more sensitivity, etc... The difference is we cant interchange them like we could with film... There are so many more factors now when evaluating cameras as it is a whole package... AF, build, weather sealing, on-board software, usability, etc... it really is a complete package rather than a single element, and this test is only testing one single element. Lastly DPR, in going back through their paces, in comparisons with nikons offerings... the 7D and the initial 5D were the first cameras they gave glowing reviews of... more or less they picked apart the xxd series to death, picked apart the 5d2 compared to the D700, and only of recent have they really warmed up to Canon, within the last 2-3 years. At least their tests are more of the camera as a whole, which I can give more weight in as a "review" site.

 

[elflord]

Your making my point. Sorry. The sensor in the d800 is not leaps and bounds over that of the D4.

I know that.
Nikon knows that.
DxO tests can't figure that out.

DxO scores of the D4 and D800 are pretty close -- basically neck and neck except for the dynamic range part of the test, hardly different by "leaps and bounds". The D4 actually does slightly better on the ISO score (2965 vs 2853)

The only substantial difference on the test is that the D800 has more dynamic range (about a stop more according to the test) at ISO100 and ISO200.

 

[jrista]

I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis

Note: lucky i cant get smited to death by the DxO brigade

I take DXO results with a nice, big, honkin grain of salt most of the time. I like the ability to include consistently-generated (hmm, grain of salt there?) low-level hardware statistics as a factor in comparing like-brand cameras (i.e. I am looking forward to seeing if they measure any hardware-level improvements between the 5D II and III), as most of the time I figure that stuff should be pretty accurate. However when it comes to some of their numbers and some of their customers (i.e. Nikon), I become more and more skeptical. With their claim of 14.4 stops of "print" DR for the Nikon D800, some 2/3rds of a stop better than the sensor itself is capable of, I've become extremely skeptical of their numbers, and usually try to have a few big, honkin grains of salt handy at all times.

When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14. DXO's DR ratings are only useful if you are shooting a scene with extreme DR to start with, and you push the "expose to the right" (ETTR) technique to the absolute limits. Even then, human error and caution will prevent you from actually achieving the maximum possible DR as indicated by DXO. The only way to actually use all of a camera's available DR is to expose a scene you simply can't expose properly no matter how much headroom it has, in which case you'll always end up with unrecoverable blocked blacks and blown highlights.

 

[iMagic]

I could care less about technical this and that. Bottom line, what are real world results preceptually (both on screen and in print) and what are the real world practicalities (useability, accessories, etc.). For example, if the final result is a print, does the DR and mega pixels really make a difference on the size/quality of print?

 

[LetTheRightLensIn]

I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis

Note: lucky i cant get smited to death by the DxO brigade

I take DXO results with a nice, big, honkin grain of salt most of the time. I like the ability to include consistently-generated (hmm, grain of salt there?) low-level hardware statistics as a factor in comparing like-brand cameras (i.e. I am looking forward to seeing if they measure any hardware-level improvements between the 5D II and III), as most of the time I figure that stuff should be pretty accurate. However when it comes to some of their numbers and some of their customers (i.e. Nikon), I become more and more skeptical. With their claim of 14.4 stops of "print" DR for the Nikon D800, some 2/3rds of a stop better than the sensor itself is capable of, I've become extremely skeptical of their numbers, and usually try to have a few big, honkin grains of salt handy at all times.

Not what you said before they posted D800 results....

When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14. DXO's DR ratings are only useful if you are shooting a scene with extreme DR to start with, and you push the "expose to the right" (ETTR) technique to the absolute limits. Even then, human error and caution will prevent you from actually achieving the maximum possible DR as indicated by DXO. The only way to actually use all of a camera's available DR is to expose a scene you simply can't expose properly no matter how much headroom it has, in which case you'll always end up with unrecoverable blocked blacks and blown highlights.

Who cares what those details are the point is to use it to compare cameras relative to one another.

 

[takoman46]

I agree that the DxO analysis does not prove that any camera is definitively superior to another camera when it comes to real world applications. It surprises me how many people gawk at the ratings and results of the DxO tests and base their purchasing decisions on them. For example, based on DxO results, the logical decision would be to purchase a D800 over a D4 or D3s even if cost was not a factor; simply because DxO tests "prove" that the D800 is superior to a D4 or D3s. Now think about it... The D4 and D800 were released in the same quarter so Nikon obviously had the opportunity to choose how they wanted the D4 and D800 sensors to be built. So Nikon puts their "greatest sensor ever" into the D800 which is not their flagship model. To me a "flagship model" defines what a company is made of and is the pinnacle of a company's achievements in research and development. So according to DxO... Nikon somehow chose not to put their "best sensor" in the D4... right... that's completely illogical and I'm sure also not true... yes, the D800 has 36 megapixels... but will it outperform a D4 in all other areas that contribute to producing a great image? I think we should ask someone who owns both a D4 and D800 to see how they feel after working with both cameras.

The same would go for Canon with the 5DmkIII and 1Dx. Although, nobody is saying that the 5DmkIII is better than a 1Dx; the 5DmkIII has more megapixels than a 1Dx. Maybe not as dramatic a jump in comparison to what's going on in the Nikon world, but the general difference still remains. Does anyone think that Canon put their "best sensor" in the 5DmkIII and put a crappier sensor in the 1Dx? What if DxO said so? Would you believe it?

Just to add support for my reasoning... (BTW, this is a non-photography related example). What if Motor Trend magazine did a test between the 2012 Chevrolet Camaro ZL1 and 2012 Corvette ZR-1 and said that the Camaro was the best sports coupe that Chevrolet ever came out with? Kind of a no brainer right? LOL

 

[briansquibb]

When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14.

The histogram in DPP seems to show DR - is that correct and accurate?

I aim for maximum DR of the shots I am taking with the 1Ds3 and max out (DPP reading) at around 10, getting mostly over 9 for iso 100/200

The equivalent for the 1D4 and 7D are about 9 max with average about 8

These are for shots that should have high DR