DXO vs Reality

[jrista]

When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14.

The histogram in DPP seems to show DR - is that correct and accurate?

I aim for maximum DR of the shots I am taking with the 1Ds3 and max out (DPP reading) at around 10, getting mostly over 9 for iso 100/200

The equivalent for the 1D4 and 7D are about 9 max with average about 8

These are for shots that should have high DR

Those sounds like entirely reasonable numbers to me...

 

[pj1974]

I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites. There are many shades of grey.

DxO has some very helpful tools and software. Just their sensor tests don't cut the mustard, imho.

Regards

Paul

Paul
since English is not my first language, sometimes it's more difficult for me to grade shades of grey and also sometimes it's more difficult to find what's thrown into one basket Anyway you clearly state what you like and dislike in DxO and this is fair. I just have problems with statements (it's not to you, Paul) I read: "they are wrong because the results are not what I expected, what I invested in, what I like, what I believe....".
It's obviously not logic because of this simple schema:
1. Did they describe their testing procedures? Yes
2. Do they follow their testing procedures during tests? Most probably, they risk too much.
3. Is there anything wrong with their testing procedures? If there is, then I'd like to read about that - why there is something wrong with the way they run their test.
4. Is there anything wrong with the interpretation of their results? Is yes, then I'd like to read about that.

I simply don't understand people, who fight against numbers, numbers having their interpretation. If someone believes there is so much wrong with DxO, then there is a simple way to proof it.
In my opinion: I do believe that numbers which DxO publishes are real. There might be sometimes something to interpretation but as far as I didn't see clearly pushing one brand over another and interpreting same or comparative results the opposite way depending on situation, then I'm not convinced they lie.

Paul, do you know what I mean?
Regards, Marek

Hi Marek

Yes, I do know what you mean. It's true that a lot of 'testing websites' meet the conditions you write about. What annoys me, is that the 'ratings' that these websites give do not always take into account many other factors (even in terms of pure image quality / sensor capabilities) - and 'not including that' - is a 'gap' - that should does not represent professionalism.

So I don't really place any of my purchasing decision on DxO's sensor tests. But they do have some useful software I'm VERY glad to have bought!

By the way, congratulations with good use of English! I speak 5 languages, so I know what it's like trying to express thoughts in a language that is not one's mother language.

Regards

Paul

 

[maxxevv]

Sorry but it's Throwing tests away is like saying: "I don't care how many HP has my engine. I don't care how fast does it accelerate. Those are just numbers." o
Of course - those are just numbers comparing if one car is faster than the other. That's about numbers as people like to see things comparable. It doesn't say what comfort is inside, nor where and how gently you drive.
Another example - I don't care if my camera/lens resolves 1000lp/mm or 400 = I don't care if my camera has 8MP or 20MP = I don't care if my camera shoots 8fps or 3fps, 14 bits TR or 8 bits TR and so on...
Let's not get crazy. Not everything is comparable but there are important things that are. Better camera will not make a photographer of anybody - that's obvious. But why not look on comparisons, tests or numbers? I don't get it.
Charts and graphs are simple enhancement to specification. What's wrong in it?

Well, there is some reality in that but also some error in the parallels.

In the context of the car, the DXO results are often more like "volume of air exhausted", "heat dissipation rate" and "150-200km/h rate".

In the grand scheme of things in practically using a car or camera, they are mostly moot ...

 

[jrista]

I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis

Note: lucky i cant get smited to death by the DxO brigade

I take DXO results with a nice, big, honkin grain of salt most of the time. I like the ability to include consistently-generated (hmm, grain of salt there?) low-level hardware statistics as a factor in comparing like-brand cameras (i.e. I am looking forward to seeing if they measure any hardware-level improvements between the 5D II and III), as most of the time I figure that stuff should be pretty accurate. However when it comes to some of their numbers and some of their customers (i.e. Nikon), I become more and more skeptical. With their claim of 14.4 stops of "print" DR for the Nikon D800, some 2/3rds of a stop better than the sensor itself is capable of, I've become extremely skeptical of their numbers, and usually try to have a few big, honkin grains of salt handy at all times.

Not what you said before they posted D800 results....

I may not have "said" it before, but I always take DXO results with a grain of salt. I think most of their statistics ARE accurate...within the context of DXO, relative to each other. I think their numbers are valuable for comparing cameras, however I have never thought their numbers should be used in isolation (I've referenced DPR results as much as I've referenced DXO results in my posts, often in the same posts). Since I may also not have said so before, let me say so now: I find DPR results, even though I believe they are more subjective and less accurate, to be just as valuable as DXO results, and in some respects, much more valuable...since they represent a real-world context better. I DID actually think their Print DR (normalized) DR results were decent before, as I fully understand the value of normalized IQ comparisons. I thought that their normalized print results, since for most cameras your downscaling...which minimizes noise and multisamples source data for each pixel, was "exhibiting the full capabilities of the native sensor DR." However I entirely expected them to come out of the gates claiming the D800 scored 13.97 on their Print DR tests, not some mysterious, magical nonsense like 14.4! I REALLY expected them to say the D800 nailed 14 stops right on the head, but instead they are basically making a claim that the D800 and only the D800 offers photographers the magical ability to GAIN ADDITIONAL DR simply by DOWNSAMPLING. Then Mt. Spokane came along posted a DXO link indicating that Nikon...but not Canon...was a big time paying supporter. Sorry, but I go where the evidence leads, and there is some evidence of very fishy behavior about DXO and Nikon these days.

For arguments sake, lets assume there is some magical gain in DR simply by downsampling. That is an ALGORITHMIC process done on DIGITIZED pixel data. The physical sensor, according to DXO's Screen DR value (which at the moment I am not suspicious of...we'll see if they claim that a future 14-bit SoNikon sensor is capable of 14.1 stops...) can produce 13.8 stops of DR strait out of the camera, no processing of any kind outside of amp and ADC. If you run into a real life scene with MORE than 13.8 stops...the theoretical possibility of using a digital algorithm to "stretch out" 14.4 stops from your RAW file isn't going to help you. A real life scene with 18 stops of DR is going to outpace even the D800 sensor, and your only option is going to be to compress the blacks into less space (and therefor less recoverability)...or use an ND filter, just like all the rest of the photographers on the great and beautiful earth.

Thanks to DXO's new D800 rating of 14.4, I am now a firm believer that Print DR is a useless measurement. Digital wizardry can not and will never be a replacement for native, hardware-level dynamic range. I now believe DXO's sole "accurate" measurement of actual hardware-level DR is their Screen DR measurement. I'm unwilling to accept Print DR measurements for any camera now as being even remotely realistic. As such, I believe the following are accurate dynamic range estimates for Nikon, Canon, and Sony cameras:

Brand	Model	DR Stops	Notes
Canon	1D IV	11.46
Canon	1Ds III	11.25
Canon	5D II	11.16
Canon	7D	11.12
Nikon	D7000	13.35	Still the best from a non-magical DR standpoint
Nikon	D800	13.23
Nikon	D4	12.58
Nikon	D90	12.21	DXO "measured ISO" closer to ISO 200, listed as ISO 200 rather than ISO 100 in chart
Nikon	D70	11.85	DXO "measured ISO" closer to ISO 200, listed as ISO 200 rather than ISO 100 in chart
Nikon	D3s	11.66	DXO "measured ISO" closer to ISO 200, listed as ISO 200 rather than ISO 100 in chart
Sony	NEX-7	12.59	Mirrorless
Sony	SLT-A77	12.35
Sony	A-900	11.5

NOTE: I was pretty sure DXO listed the Screen DR for the D800 at 13.8 stops before. It is currently listed as 13.23...which makes their claim of 14.4 stops "Print DR" even more insane. That would be 1.17 stops of MAGICAL DR gained by the simple act of DOWNSAMPLING?!?!? BULL SH*T! BIG, STEAMING PILE OF BULL SH*T!

Print DR == worthless tool for comparison.

Given the table above, briansquibb's numbers of a about 10 stops of real-world DR for his 1DsIII seem pretty spot on. It would be very difficult to actually utilize the full 11.25 stops without at least blowing out one color channel. Its tough to determine how much shadow DR you might be capturing as well, since you can't really "clip" shadows...you just might not actually gather enough light to measure as a reading in a pixel. Real-world 10-stop DR with an 11.25 stop camera seems entirely valid and realistic to me. I would suspect something similar with the D800...you might get about 12-stop real-world DR with the 13.23 stop megapixel flagship camera if you really pushed for it.

 

[KeithR]

Having owned a Nikon D70 waaaay back in the day (my first DSLR - I've still got it somewhere, because it'll get me nothing if I sell it) I can say with absolute confidence that the notion that this camera has better Real World DR than my current Canon 7D is utterly, utterly risible.

So much for DxO, then.

 

[briansquibb]

Having owned a Nikon D70 waaaay back in the day (my first DSLR - I've still got it somewhere, because it'll get me nothing if I sell it) I can say with absolute confidence that the notion that this camera has better Real World DR than my current Canon 7D is utterly, utterly risible.

So much for DxO, then.

+1 Not convinced the 1D4 is better than the 1Ds3 either

 

[jrista]

Having owned a Nikon D70 waaaay back in the day (my first DSLR - I've still got it somewhere, because it'll get me nothing if I sell it) I can say with absolute confidence that the notion that this camera has better Real World DR than my current Canon 7D is utterly, utterly risible.

So much for DxO, then.

Having owned a Nikon D70 waaaay back in the day (my first DSLR - I've still got it somewhere, because it'll get me nothing if I sell it) I can say with absolute confidence that the notion that this camera has better Real World DR than my current Canon 7D is utterly, utterly risible.

So much for DxO, then.

+1 Not convinced the 1D4 is better than the 1Ds3 either

It would depend on what your observing. First off, most "good" computer screens these days are 8-bit screens, and the best of the best are 10-bit screens (which require a matching 10-bit video card to fully realize). That would mean that on a computer screen, the best you could "observe" in terms of DR is about 8-10 stops worth. The bastion of additional dynamic range is not really in what you can see, its in how far you can push exposure and what you can recover. I'd be willing to bet the 7D has more highlight headroom than the D70, and I'd be willing to bet the same thing about the 1D IV vs. the 1DsIII. Canon does kind of suck on the shadow DR end of things, but I've been able to push their exposures extremely far to the right (compress the highlights) such that when I have a histogram that is riding up the right-hand side and would seem to be fully blown, I can usually recover everything...and at worst, I might blow out a tiny bit of one color channel. My Canon 7D has more highlight headroom than my 450D did for sure (which according to DXO Screen DR has about 10.47 stops of native DR), and I've noticed much greater highlight recovery when exposing the moon or a sunset than I did before.

 

[briansquibb]

It would depend on what your observing. First off, most "good" computer screens these days are 8-bit screens, and the best of the best are 10-bit screens (which require a matching 10-bit video card to fully realize). That would mean that on a computer screen, the best you could "observe" in terms of DR is about 8-10 stops worth. The bastion of additional dynamic range is not really in what you can see, its in how far you can push exposure and what you can recover. I'd be willing to bet the 7D has more highlight headroom than the D70, and I'd be willing to bet the same thing about the 1D IV vs. the 1DsIII. Canon does kind of suck on the shadow DR end of things, but I've been able to push their exposures extremely far to the right (compress the highlights) such that when I have a histogram that is riding up the right-hand side and would seem to be fully blown, I can usually recover everything...and at worst, I might blow out a tiny bit of one color channel. My Canon 7D has more highlight headroom than my 450D did for sure (which according to DXO Screen DR has about 10.47 stops of native DR), and I've noticed much greater highlight recovery when exposing the moon or a sunset than I did before.
[/quote]

I only use the histogram in DPP. I find the 1Ds3 has about +/- 5 so it meters in the middle - which I am happy about as it leaves room for level adjustment

 

[dichiaras]

Can everybody settle down?
Canon, either with the crop sensor or the full frame one, has less DR than Nikon's corresponding sensors: everywhere you look, DPR, DxO, or individual photographer's reviews, that's what you get. It's a fact of life, so live with it!

On the other hand, even if I had a Nikon D800 instead than my very modest Canon 1100D (which I would love to get rid of once a new Canon crop sensor comes out), my pictures wouldn't get noticeably better, just because very very rarely I need all that DR, color depth, and ISO performance that Nikons offer. And I'm sure neither does 99% of all of you writing on this forum (and 95% of those just reading it, since they spend better their time by actually taking pictures )...

SO JUST CALM DOWN

 

[jrista]

It would depend on what your observing. First off, most "good" computer screens these days are 8-bit screens, and the best of the best are 10-bit screens (which require a matching 10-bit video card to fully realize). That would mean that on a computer screen, the best you could "observe" in terms of DR is about 8-10 stops worth. The bastion of additional dynamic range is not really in what you can see, its in how far you can push exposure and what you can recover. I'd be willing to bet the 7D has more highlight headroom than the D70, and I'd be willing to bet the same thing about the 1D IV vs. the 1DsIII. Canon does kind of suck on the shadow DR end of things, but I've been able to push their exposures extremely far to the right (compress the highlights) such that when I have a histogram that is riding up the right-hand side and would seem to be fully blown, I can usually recover everything...and at worst, I might blow out a tiny bit of one color channel. My Canon 7D has more highlight headroom than my 450D did for sure (which according to DXO Screen DR has about 10.47 stops of native DR), and I've noticed much greater highlight recovery when exposing the moon or a sunset than I did before.

I only use the histogram in DPP. I find the 1Ds3 has about +/- 5 so it meters in the middle - which I am happy about as it leaves room for level adjustment

I'll have to poke around with DPP, see what it says about my cameras. I wonder if Nikon has any similar software that offers the same kind of DR insight.

 

[briansquibb]

On the other hand, even if I had a Nikon D800 instead than my very modest Canon 1100D (which I would love to get rid of once a new Canon crop sensor comes out), my pictures wouldn't get noticeably better, just because very very rarely I need all that DR, color depth, and ISO performance that Nikons offer. And I'm sure neither does 99% of all of you writing on this forum (and 95% of those just reading it, since they spend better their time by actually taking pictures )...

SO JUST CALM DOWN

I dont think you will find that in real life that Nikon's are appreciably better than Canon, regardless of the 'scientific' reviews that go on. The only thing that matters is is the resultant output.

I find that I do need the maximum dr for my photos - however it is more important to use the right techniques to get that from the equipment you have rather than the 'other' brand.

I have no interest in switching to Nikon - it would cost a lot and the gain would be, if anything, very small in Reality. This thread is examining the DXO measurement system which for some does influence peoples choice of body. However as you will have noticed that the opinion is swinging such that there is little conidence in the DXO reviews.

I dont think this thread has got at all heated

 

[davidpeter]

Don't take it as an offense, but this whole topic shows that you don't have a clue about the methods and results of DxO testing.
DxO is very accurate and reliable, and bench marking sensors is only the showcase of the lab's work. But anyone, who does not posses some advanced knowledge about AD converters, Fourier transform, digital imaging and such things should not care them. You only fool yourself, when you compare numbers without knowing their meanings. Worse, when you argue about numbers without knowing their meaning, you fool others too.

For me, DxO is one of the primary factors when I'm buying a new body as I know how to handle the scores. The other primary factors are the test pictures, what I took myself. This pair has never failed me.

 

[LetTheRightLensIn]

However I entirely expected them to come out of the gates claiming the D800 scored 13.97 on their Print DR tests, not some mysterious, magical nonsense like 14.4! I REALLY expected them to say the D800 nailed 14 stops right on the head, but instead they are basically making a claim that the D800 and only the D800 offers photographers the magical ability to GAIN ADDITIONAL DR simply by DOWNSAMPLING. Then Mt. Spokane came along posted a DXO link indicating that Nikon...but not Canon...was a big time paying supporter. Sorry, but I go where the evidence leads, and there is some evidence of very fishy behavior about DXO and Nikon these days.

Enough with the area 51 stuff ;D they 'magically' make ALL of the cameras that have more than 8MP do better, not just the D800 and not just Nikons, note the 5D2 is like 11.2 unless you look at the print plot and only then does it hit the 11.8-11.9. No black helicopters here. ;D

If you run into a real life scene with MORE than 13.8 stops...the theoretical possibility of using a digital algorithm to "stretch out" 14.4 stops from your RAW file isn't going to help you. A real life scene with 18 stops of DR is going to outpace even the D800 sensor, and your only option is going to be to compress the blacks into less space (and therefor less recoverability)...or use an ND filter, just like all the rest of the photographers on the great and beautiful earth.

a print viewed from farther away might seem to have better DR
anyway as we've both said the absolute numbers don't even matter, it's the relative numbers that matter

Thanks to DXO's new D800 rating of 14.4, I am now a firm believer that Print DR is a useless measurement.

No it is not, it is the one that makes sense since it normalizes things. Without it you'd penalize a D800 vs a 5D AND a 5D2 vs a 5D etc.

Digital wizardry can not and will never be a replacement for native, hardware-level dynamic range. I now believe DXO's sole "accurate" measurement of actual hardware-level DR is their Screen DR measurement. I'm unwilling to accept Print DR measurements for any camera now as being even remotely realistic. As such, I believe the following are accurate dynamic range estimates for Nikon, Canon, and Sony cameras:

no the screen may make sense for looking at an individual camera or even for seeing how you do when you maintain full res of each, but it is not a fair way to compare different cameras in general at all

 

[LetTheRightLensIn]

Having owned a Nikon D70 waaaay back in the day (my first DSLR - I've still got it somewhere, because it'll get me nothing if I sell it) I can say with absolute confidence that the notion that this camera has better Real World DR than my current Canon 7D is utterly, utterly risible.

So much for DxO, then.

it doesn't which is why jrista is totally wrong claiming that the screen measurement is the one to use, using the print measurement, as you should, it no longer has the stop advantage

 

[jrista]

But anyone, who does not posses some advanced knowledge about AD converters, Fourier transform, digital imaging and such things should not care them. You only fool yourself, when you compare numbers without knowing their meanings. Worse, when you argue about numbers without knowing their meaning, you fool others too.

That, right there, pretty much sums up the problem with DXO quite perfectly: The exact meaning of their numbers are not fully known!! Thats my entire complaint about Print DR...we don't know what the hell it really is, exactly how it is derived, and exactly what mathematical and procedural algorithms are applied to images when scaling to their 8x12 print size. Worse yet, we don't know if they use a consistent approach to that process, and with the results of the D800, whether DXO uses a "consistent approach" is exactly why they are in the crosshairs.

Now, I personally do understand how ADC's, fourier transforms, digital imaging, color processing and color spaces, color fidelity and color space conversions, spatial frequencies/MTF, and the physical nature of light work, and I know it pretty damn well. I may not be a DXO Labs "scientist", but I do understand how those things work. Despite that, I STILL find the numbers DXO produces to be useful ONLY WITHIN the context of DXO. Without explicit details about how every one of their tests is done (we only have vague details and rolled up mathematical formulas, etc.), DXO numbers don't really mean anything in the real world.

So yeah, no one is really truly qualified to use DXO numbers in any meaningful context...and that is also exactly the problem, especially when people put too much weight on their results. They may be consistent, but if D800 Print DR is telling to any degree, they are seeming more and more to be consistently useless.