Mother of God - D800 scores 95 DxOMark

[sarangiman]

Ok, I get you, Neuro. Yeah I pay no attention whatsoever to DXO's 'Sensor Score'.

2. by bias I am saying the assumption/pre-condition DXO sets up might be biased. In other words, Nikon un-intentionally take advantage of it.

That's an interesting conspiracy theory and one, I must admit, I didn't discount earlier on much like yourself. But, if this were true, then:

• Nikon shadows raised in post would appear blurry due to noise reduction. They do not.
• Whatever processing you suspect Nikon DSP may be doing can be replicated on the RAW Canon file in post. Clipping, noise reduction, you name it. Varying the clipping point (e.g. 2047 for 5D3) doesn't change the point at which the response curve on a log scale 'toes' (bottom of hockey stick), so that irrelevant IMHO. It does 'clean up' the shadows by making the signal go to 0 (black) quicker and the noise drop (standard deviation goes from 5 to 1.4 for the darkest patch as you go from 2047 to 2057 for the black clipping point; I wrote some code to quickly analyze these types of questions so I've tried a number of things but cannot replicate the cleanliness of the D7000 dark patches)... but that doesn't magically increase the SNR or change the point at which the curve toes. Meanwhile, the darkest patch on the D7000 still has SNR>1. Maybe what we could do is apply noise reduction to shadows on the Canon file to see if we can better SNR in those dark patches; however, again, I don't think that's magically going to change the point at which the curve toes (i.e., the point at which the sensor no longer accurately record the scene luminance changes). But I'll try this in my analysis just to ease such concerns.

3. I agree with you on the D7000 low noise performance, although I don't know if it comes from the circuit or the DSP. I doubt Canon can not do this. They just don't do as Nikon. Canon's approach seems different and hurts the result at low light which is not appreciated by me.

Like I said, I doubt this to be true b/c if it were then with enough image processing you'd be able to make the Canon shadows look like the Nikon shadows. Fred Miranda's, or any of a number of other reviewers', comparison demonstrate that there's just no way in heck you could clean up the Canon shadows to get the levels of detail/cleanliness present in the D800. You can't magically create detail. If the transitions from patches 35-42 on my 13.2 stop wedge are completely lost in noise in the Canon file, you can't magically get those transitions back with noise reduction (i.e. create detail).

But I guess to convince you I could try heavy noise reduction & then quantitation to see if it helps mitigates the deviation from the model (of what patches 35-42 should be). I can almost guarantee right now it won't.

[Marsu42]

IQ deduced from a sensor score that reflects numerous measurements not relating to IQ?

Hugh? Even dxo's critics in this thread state that the measurements aren't the problem, just the non-transparent computation of the final score.

Google's algorithm does not give a score that influences spending habits nor does it pretend to be 100% accurate.

You should think again about this statement: Google does give a score (rank 1, 2, 3, 4) and if you don't think the Google rank influences spending habits ... well, what can I possibly say?

Where does dxo state that their sensor score is 100% accurate and the ultimate judgement? Every smart person will know a overall score is a reduction of complexity and thus has to lose accuracy - so it's not wise to take it as a sole basis for a decision. And while it is regrettable dxo doesn't disclose their formula, this fact alone doesn't prove they're out to get Canon.

[briansquibb]

IQ deduced from a sensor score that reflects numerous measurements not relating to IQ?

Hugh? Even dxo's critics in this thread state that the measurements aren't the problem, just the non-transparent computation of the final score.

Google's algorithm does not give a score that influences spending habits nor does it pretend to be 100% accurate.

You should think again about this statement: Google does give a score (rank 1, 2, 3, 4) and if you don't think the Google rank influences spending habits ... well, what can I possibly say?

Where does dxo state that their sensor score is 100% accurate and the ultimate judgement? Every smart person will know a overall score is a reduction of complexity and thus has to lose accuracy - so it's not wise to take it as a sole basis for a decision. And while it is regrettable dxo doesn't disclose their formula, this fact alone doesn't prove they're out to get Canon.

I have never said DxO are out to get Canon

My Google doesn't give a rank. just a long list of sites. Google doesn't influence spending habits - does Google say 'this product is better than that product'?

Unlike DxO which gives comparisons based on unknown weighting and algorithms

[dilbert]

So as far as I'm concerned, DxO's dynamic range numbers are absolutely valid.

I agree.  Allow me to clarify.  I see no issues with DxOMark's measurements - they are useful and accurate, and they disclose their methods to generate those measurements. (FWIW, I personally prefer and use DxO Optics Pro as a RAW converter.)

The problem is the DxOMark Sensor Score.  Here's what Luminous Landscape states:

Documentation about the way the final DxOMark Sensor score is computed from Dynamic Range, Color Sensitivity and Low-light ISO scores is not currently available.

That same article (linked in a thread started by dilbert) indicates there is some 'non-linearity' in the way the Score is calculated.  So...a weighted average, but undisclosed weighting factors, no confirmation that those unknown factors are even consistently applied, non-linear calculations, etc., to me, that makes the DxOMark Sensor Score a meaningless and useless number (not measurement, just an irrelevant number).

So for me, the DxO sensor score for the 5D Mark III seems to be rather close to this:

(portrait color depth bits) + (landscape dynamic range) + sqrt(sports low light ISO)
24                                                         11                                47
= 82

their score is 81 and I've rounded down (or rather truncated) two of the above numbers.

For the D800, I get 25+14+39 = 92

But using the same measure, the D3200 would only be 70 so there's obviously some compensation in their scoring for crop sensors. For the Canon G1X, I get 21+10+25=56 (not 60).

Keep in mind that I'm just making guesses here about what sort of transform they use to end up with the DxO sensor score.

[Marsu42]

I have never said DxO are out to get Canon

And I have never said you said it.

My Google doesn't give a rank. just a long list of sites. Google doesn't influence spending habits - does Google say 'this product is better than that product'?

Omg - please catch up with digital reality, will you? As my one, first and last hint on this: the product seo companies sell to their corporate clients is called a rank, and it's much more important than any test score because it *does* influence purchases like nothing else. If Joe Sixpack types "video dlsr" in Google, and the first page is filled with Nikon products, what will he think is best and what will he buy, hmmmmm?

Influencing people doesn't need a direct comparison like a numerical score, or advertising wouldn't have any sense at all. At least with an aggregated score, smart people will know it's subjective, unlike Google ranking ("long list of sites") which seems to be valued too low my many - obviously including you.

[elflord]

My Google doesn't give a rank. just a long list of sites. Google doesn't influence spending habits - does Google say 'this product is better than that product'?

They don't have to say anything. Where someone appears on that list will affect a websites hit count.

If it were possible to simply "buy" a ranking on the google search engines, there would be a market for it and people would pay for it.

Unlike DxO which gives comparisons based on unknown weighting and algorithms

The aggregate score is clearly subjective, because the choice of how to weigh the factors is subjective. We should take the aggregate score for what it is -- DxO's  subjective take.

However, there have not just been complaints about the aggregate sensor scores in these threads, but also largely unsubstantiated attacks on DxO, originating from those who don't like the numbers.

The measurements are what they are. It's not DxO's fault that the data are what they are.

[loveboxer]

After my partner and I did our side by side tests, we realized that we could no longer refer to DXO's findings in making any final decisions, whether DXO is accurate or not.

The simple reason is that when we presented approximately 25 portraits shot side by side with the 5D3 and D800 using the exact same settings and positioning, the Canon shot was chosen 95% of the time.

I want to emphasize that I am simply sharing my findings, I do not idolize either company and because of habit am better acquainted to Nikons personally.

I love both companies and full heartedly believe that beautiful photography will never be the result of a camera, it is from the artist.

[sarangiman]

However, there have not just been complaints about the aggregate sensor scores in these threads, but also largely unsubstantiated attacks on DxO, originating from those who don't like the numbers.

The measurements are what they are. It's not DxO's fault that the data are what they are.

+1. Exactly.

The simple reason is that when we presented approximately 25 portraits shot side by side with the 5D3 and D800 using the exact same settings and positioning, the Canon shot was chosen 95% of the time.

Did you shoot JPEG?

[Marsu42]

ere possible to simply "buy" a ranking on the google search engines, there would be a market for it and people would pay for it.

This is off-topic, but fyi: There is a flourishing market for this, it's called seo ("s3arch engine optimization") and many many blogs are run for the sole reason to put client's links for money on them, boosting the Google rank of the linked site... a friend of mine is working full-time building these pseudo-sites, writing or letting write blog posts while then adding a paid article with a link. With this system, Google cannot tell a real blog site from a seo one, though they keep trying.

[Ellen Schmidtee]

The D800 has 14 bits A/D converters, so how could it have more than 14 bits of dynamic range?

A JPEG file is 8-bit, but that doesn't mean that the maximum scene dynamic range it can encode is only 8 stops, correct?

I'm not so sure.

Think about printing greyscale images with only pure black ink. So you basically have only two colors pure white and pure black, one stop DR only? And it is possible to print a fine greyscale image if the resolution is fine enough. The technique is some kind old.

OK, I'll try to be more explicit.

As far as I understand, the photosites measure the amount of light falling on them, and the A/D converter turns the analog energy measurement into a number, and those two stage are linear. Therefore, no matter how many shades of gray are there between 2^14 and 2^13, those two shades are one stop apart.

Do I misunderstand?

[elflord]

(portrait color depth bits) + (landscape dynamic range) + sqrt(sports low light ISO)
24                                                         11                                47
= 82

I don't think sqrt(low light ISO) makes sense -- you need to take the log of it. The other two scales are in stops, so the third score needs to be transformed to get it into stops. You might need to transform it to get it right after that (e.g. add an offset and a multiplier).

The way you're doing it with sqrt instead of log, the function will grow too quickly (which is why you get lower scores for small sensor cameras)

[dilbert]

(portrait color depth bits) + (landscape dynamic range) + sqrt(sports low light ISO)
24                                                         11                                47
= 82

I don't think sqrt(low light ISO) makes sense -- you need to take the log of it. The other two scales are in stops, so the third score needs to be transformed to get it into stops. You might need to transform it to get it right after that (e.g. add an offset and a multiplier).

The way you're doing it with sqrt instead of log, the function will grow too quickly (which is why you get lower scores for small sensor cameras)

I tried log but then those three numbers alone did not work (the numbers were too small.)

But it could be that it is "portrait+landscape+log(lowlight)+something_else".

[elflord]

I tried log but then those three numbers alone did not work (the numbers were too small.)

But it could be that it is "portrait+landscape+log(lowlight)+something_else".

You need log_a( b*score)

or equivalently,  B+ log(score)*A.   

[lonebear]

To Ellen Schmidtee,

Your are correct, but that's the DR of a single pixel, not the whole picture. When pixels are fine enough and closing to each other enough, human eyes tend to blend them together. In that case, the DR is not based on a single pixel but a cluster of pixels.

[dilbert]

I tried log but then those three numbers alone did not work (the numbers were too small.)

But it could be that it is "portrait+landscape+log(lowlight)+something_else".

You need log_a( b*score)

or equivalently,  B+ log(score)*A.

So far as I could tell, b is not constant.

Given that, I'm willing to believe that the idea here is close enough to what DxO do  to give people an idea for the score as being objective. They may, for example, be using SNR readings for the graph or maybe the SNR is part of the lowlight ISO score or .... they're unlikely to tell us what they do because it is their product (the DxO score) and publishing the formula would allow anyone to publish a DxO score.