Can anyone explain what on Earth DXOMark is on about?

[J.R.]

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

+1 ;D ;D ;D

 

[Loswr]

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

Show me where their sensor measurements include sharpness as a tested parameter. Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg. You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions. The problem is your false assumptions, not the reported data.

 

[traveller]

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

Show me where their sensor measurements include sharpness as a tested parameter. Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg. You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions. The problem is your false assumptions, not the reported data.

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores. [DXOMark]

Is DXOMark confusing the issue, or are they just confusing me! ???

 

[J.R.]

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

Show me where their sensor measurements include sharpness as a tested parameter. Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg. You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions. The problem is your false assumptions, not the reported data.

Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

 

[mitchell3417]

All this does it put numbers to what we already knew. When you put a lens on a D800 and a 5DIII and go out and take pictures the difference in IQ (specifically resolution in this case) is not as large as some people would have you believe. Both cameras are great. It's not worth arguing over. The only reason one needs to compare cameras is if one is going to invest in a new camera system. In that case, one should rent the cameras in question and decide for himself which one best fits his needs.

 

[Loswr]

Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores. [DXOMark]

Is DXOMark confusing the issue, or are they just confusing me! ???

I think it's them, you have a point about the article itself. They start off and conclude with what I think is the central point of the article, illustrated by these data (as a side note, I'd love it if this plot was 'live' i.e. you could click a data point and see the lens it represents):

But, after presenting that central piece of information, they felt compelled to discuss why that differs somewhat if instead of just looking at sharpness (P-Mpix), you look at their summary 'DxOMark Lens Scores' (and you may recall how I feel about their Overall Sensor Score...the Lens Score not really any better). But still, they could have simply stated, "The above comparison is in regards to sharpness on the 5DIII vs. D800, and the Lens Score considers parameters other than sharpness, including low light performance on a given body," and been done with that bit.

 

[Mt Spokane Photography]

One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.

They have started testing lenses a while back, and publishing results in conjunction with DPR. I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.

We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%. This is due to a lens resolution, circle of confusion, and other factors. When you get high pixel densities, resolution of the system does not scale, but is always a improvement.

That is what DXO mark is saying.

Technically, the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link. Even film has MTF values specified.

So, for simplicity, if a lens has a MTF of 0.9, a Body 0.8, together they are 0.72. Increase the body to 0.85, and the system becomes 0.765. Better, but not a revolution because the lens needs to get better as well.

However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).

One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.

 

[Aglet]

One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.

They have started testing lenses a while back, and publishing results in conjunction with DPR. I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.

We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%. This is due to a lens resolution, circle of confusion, and other factors. When you get high pixel densities, resolution of the system does not scale, but is always a improvement.

That is what DXO mark is saying.

Technically, the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link. Even film has MTF values specified.

So, for simplicity, if a lens has a MTF of 0.9, a Body 0.8, together they are 0.72. Increase the body to 0.85, and the system becomes 0.765. Better, but not a revolution because the lens needs to get better as well.

However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).

One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.

That's about right

Or, what DxOmark's testing translates to in this regard is simply

DIMINISHING RETURNS of increasing sensor resolution

Adding 60% more MP does not give correspondingly (square root of ratio) more resolution.

Adding more MP can increase overall acuity, since it's part of an equation containing the factors mentioned above, but it's only one factor.

another good example is in DPreview's comparison of the AA-less D7100 and the AA-equipped D5200, both with very similar sensors of equivalent pixel count.

www.dpreview.com/previews/nikon-d7100/6

only in the small range of optimal lens performance (wider apertures) can a difference be noticed whereas regular shooting with real-world lenses and settings this AA-less sensor performance is no better.
(personally I'd take the d5200 over the 7100 for better performance/cost, slightly lower overall sensor noise and it oddly fit my hand better than the 7100)

 

[Albi86]

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

I'm not disturbed by that because I'm aware it's an approximation deriving from their test methodology. In fact I'm not very interested in the absolute value, but instead I find it relevant in relative terms - I.e. camera x vs camera y.

What I find disturbing is the vaguely (yet apparent, at least to me) apologetic tone of this article. Examples:

5D Mark III has a highly sensitive and accurate 61-point focusing system, durable pro-level build, excellent live view implementation, and can capture stills at up to 6fps (compared with 4.5 fps for the Nikon).

When did they start taking such things into account?

When using specific lenses (such as the new Sigma 35mm f/1.4 DG HSM A) the Nikon can out resolve the Canon. However, taken as a whole, the statistics reveal the EOS 5D MK III is capable of similar sharpness and of achieving a close DxOMark camera/lens score to the Nikon D800. Moreover, that’s despite the latter camera’s 60% extra pixel count.

So... Are we comparing cameras, or camera systems, or what? Because saying that a 5D3 with a good lens is better than a D800 with a crappy lens doesn't sound like a great revelation to me.

While that same principle of choosing the best glass also applies to the Canon, it’s not quite as crucial. In financial terms alone, this may be an important consideration if you have already invested in a sizeable range of lenses. Lenses on the Canon EOS 5D Mark III are without doubt very efficient.

Why not keeping the old D700 then? Even less demanding with only 12 mp, and it also makes you save some grands on a new camera!

Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well.

Here a comparison with the D600/D700 would have been nice.

If you’re a Canon user and are looking to upgrade, or maybe even switch because of the sensor’s perceived lowly capabilities then this should put your mind at rest.

That's where they wanted to get. All the rest IMHO is a filler.

So yes, I preferred when they just tested sensor performance. That's the only thing that can be done in a scientific and easily comparable way. Adopting this sterile approach on more subjective matters is just misleading IMHO. And this article feels very bizarre to me in its style, content and conclusions.

 

[David Hull]

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

Yep... I think it really just comes down to that. People need to think in terms of the system and not just the individual components. That is probably the only take away from all of this.

I found it interesting that with the so-called "Holy Trinity" (which is what most pros would be packing or at least lusting after) i.e. the 16-35L II, the 24-70L II and the 70-200LII, produces better performance when combined with the 5DIII than the Nikon equivalents do when combined with the D800.

 

[gary]

Having read the parameters for testing I have to say I am more than a little confused. Comparing apples and oranges is not very scientific unless of course you are testing to see that an orange tastes like an orange. As a camera doesn't really function well without a lens I have tended to ignore pure sensor tests so was intrigued to see a test of bodies and lens. I was then sadly let down when I didn't see side by side like for like comparisons. I think I will just stick to looking at the photo's I take and make my own judgement.

 

[Sporgon]

Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!

That's OK. If I may quote Benedick from 'Much Ado About Nothing'

"In a false quarrel there is no true valour" ;D

 

[Loswr]

with a good lens, and there are many, high resolution will allways be better than less.

Of course. But the point of the DxOMark wasn't 'with a good lens' it was an average across a group of lenses.

I do find it quite interesting that for some common lenses like the 'workhorse' 24-70 f/2.8, FF 'kit' 24-105/120 f/4, and 100/105 Macro, for sharpness the Canon lens on the 5DIII outperforms the respective Nikon lens on the D800. The first two pairs are 'general purpose zooms' and are arguably some of the most commonly used lenses, the 3rd pair are the most common macro lenses. In those cases, with these common lenses, the D800 offers no resolution advantage over the 5DIII.

 

[jthomson]

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

What disturbs me is the lack of apples to apples comparisons. It would have been good if they had at least given some comparisons of the two camera's with the same lens. If I'm reading the data correctly the Canon 5D3 with the Sigma 85 f1.4 has the same DxO score as the D800 with the same lens. The data give the sharpness of the Canon combo in their top ten, but not the sharpness of the Nikon Combo. Does this mean the Canon combo is sharper that the nikon combo or did they just not test the nikon combo. All their top ten sharpness combo's tell me is that the Canon 300mm f2.8L is one great piece of glass.

It would also have been informative if they had tested some of the nikon lenses on Canon bodies. What happens with a nikkor 85mm f1.4 G on a 5D3.

 

[Pi]

DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores.

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score. It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value. Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement. That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality).

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.

 

[Loswr]

well , it must be some errors

As far as I know, DxO tests just one copy of a lens. Given Sigma's reputed less-than-stellar QC (not than Canon/Nikon are perfect, far from it). So, perhaps not errors on DxO's part, but rather copy variability as seen in Roger Cicala's testing of his rental stock.

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.

I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).

 

[Pi]

I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).

At what distance, and what print size?

 

[Loswr]

At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height). Granted, there's no guarantee they're following convention...

 

[Pi]

At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height). Granted, there's no guarantee they're following convention...

What does "∝" stand for? If it means "equal", then this depends on the units (m, cm, ft) used. If means "proportional", then SQF depends on the constant. In all cases, there is some choice of distance/print size incorporated.

The way it is calculated, it is some kind of weighted average of the MTF values. Well, what is wrong with those values in the first place as a way to report what they measure? Any attempt to represent a whole function by one number is a failure.

 

[Loswr]

What does "∝" stand for? If it means "equal", then this depends on the units (m, cm, ft) used. If means "proportional", then SQF depends on the constant.

The ∝ symbol is mathematical shorthand for 'is directly proportional to'. The full relationship (again, by convention) for viewing distance (d) and picture height (PH) is: d = 30cm x √(PH /10).

I do agree that representing a complex optical system with a single number is over simplifying. Have you seen Bob Atkins' illustration of that concept? He makes the point quite well with the analogy of representing an image as its average color: