Can anyone explain what on Earth DXOMark is on about?

[traveller]

This has to be DXOMark's most bizzarely written article ever (and it has a lot of competition!)

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected

 

[Albi86]

This has to be DXOMark's most bizzarely written article ever (and it has a lot of competition!)

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected

Weird. It basically contradicts everything they've been publishing so far. Maybe they are a bit concerned about loosing popularity among Canon users and are stepping back?

 

[Loswr]

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

 

[Sporgon]

Maybe they have come to realise that they are leaving the real world behind.

Obviously a lot of work goes into their 'testing', but they don't do themselves any favours IMO, from lumping contradictory factors together and producing an 'average' score.

Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

 

[Loswr]

Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.

 

[traveller]

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

"In future tests, it will be interesting to see if the Sony sourced sensor in the Nikon D800E variant with its altered (zero strength) OLPF (Optical Low-Pass Filter) is significantly more efficient at resolving detail or if it’s as a result of the differences in fill-factor (affected by RGB filter transmission, micro-lens design and circuitry) between the Canon and Nikon sensors.

Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well." [DXOMark]

I think they're saying more than that; it seems that they're implying that there is a difference in the architecure of the 5D Mark III sensor that means it resolves more detail 'per pixel' than either the D800 or previous Canon full frame sensors.

 

[Albi86]

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

If you compare a 3rd party lens - let's take the Zeiss 100 MP, which is known to be pretty sharp - on 5D3 and D800, they rate it sharper on the 5D3 - which makes no sense. However, its total score is lower than if combined with the D800 - which also makes no sense.

However, one explanation can be:

Sharpness is a subjective quality attribute of an image or a lens. Sharpness indicates the visually perceived quality of details of an image or details reproduced by a lens. It is associated with both resolution and contrast of reproduced details (within an image or by a lens).
The DxOMark score for Sharpness is based on the Perceptual Megapixel (P-Mpix) concept that weights the Modulation Transfer Function (MTF) of the lens with the human visual acuity. Read more about Perceptual Megapixels.

If they take contrast and thus "subjective perceptions" into account, it all looses any scientific relevance. They were better off measuring sensor parameters alone.

 

[Loswr]

While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.

 

[cocopop05]

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said.

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings.

 

[Albi86]

While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.

I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

It's simple. The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least). The resolution we care about is the system performance = sensor + lens. So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary). I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said.

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings.

DxO is not a public service or a no-profit. They are a company, and as such they have all the interest in publishing things that will increase their popularity among users of the market leader company.

 

[Pi]

I think they fall victim of their own undocumented P-Mpix metric. Some of the claims are absurd, or trivial. I have seen many comparisons (for example, on IR) which show the D800 resolving more than the 5D3 without any doubt, with lenses which are not necessarily state of the art.

"Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well." - I m surprised that they are surprised. The 7D is not resolving as much as the 350D pixel by pixel but resolves more on image level. Why would anybody be surprised by that?

 

[Sporgon]

Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.

OK, well perhaps it would be more relevant to suggest that this test may have been prompted by Roger at Lens Rental's comparison between the D800 and 5D with various standard zooms - reported here on CR.

Anyway, within sensors of the same size, would you not expect one with larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?

 

[Loswr]

I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

Results will always depend on what's being measured. Roger uses Imatest and presents MTF50 data. That's a useful comparator benchmark in some ways, but a poor one in others. DxOMark now uses P-Mpix, which considers acutance and contrast in addition to MTF, and better approximates human perception of sharpness. It's worth noting that DxO's P-MPix is similar to the Subjective Quality Factor (SQF) that Imatest incorporated a few years ago (the concept of SQF has been around for 40 years), for the same reason - MTF50 doesn't tell the whole story when it comes to sharpness .

 

[Albi86]

Anyway, within sensors of the same size, would you not expect one with larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?

Yes, this is why the 5D2 also scores pretty well in this regard. Similar pixel size to the 5D3 but weaker AA filter, as far as I know.

What's disturbing about this article is the way results are presented.

 

[TrumpetPower!]

If I read them right, they're basically claiming that all those extra megapickles of the D800 just get you larger files, but they don't get you any sharper prints.

I can believe that without trouble.

Consider all those who've compared 1.6x cropped images from the 5DIII with the 7D and found the 5DIII to be the clear winner. That's the exact same type of comparison, only with an even more dramatic difference in mepickle count.

If DXO's numbers are are all meaningful, then they should discover the 1DX absolutely trouncing the D800 on their sharpies-per-megapickle score. If they don't discover that, then this is nothing more than yet another bizarro-world imaginary number pulled from their collective netherbits.

I gotta say, though, this is another one of those bizarre "My minivan has a top speed of 110 mph but yours has a limiter that kicks in at 95 mph, so neener neener" types of things. You can already make absolutely stunning 24" x 36" prints from any of these full-frame cameras with ease, and you can max out the width of a 44" printer with a single exposure with care so long as viewers won't actually physically be sticking their noses in them. And for what those kinds of prints sell for, if you really need to deliver them with any type of regularity, you can easily afford the medium format gear that's actually designed to do that sort of thing.

I'm not at all exaggerating when I state that a 5DIII with a TS-E 24 has significantly more image quality than anything Ansel Adams ever laid hands on, including his 8" x 10" view camera.

So, while more is awesome, can't we be happy with what we've already got?

Cheers,

b&

 

[Rienzphotoz]

Interesting :-\

 

[traveller]

I think they fall victim of their own undocumented P-Mpix metric....

I think you're probably right; GIGO, as they say!

DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores.

 

[Loswr]

DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores.

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score. It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value. Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement. That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality).

What's disturbing about this article is the way results are presented.

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

 

[Sporgon]

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

 

[TrumpetPower!]

I'm curious...what is disturbing about the way the results are presented? Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800? :

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... : :

Or, in other words, if it was a cheese shop, it wouldn't be much of one, would it, now -- even if it was the finest one in the lineup. So clean, really -- in the sense, of course, that it's uncontaminated by photos.

b&