DXO finally re-tests EF lenses on the 5DS R

[ahsanford]

This just dropped:
http://www.dxomark.com/Cameras/Canon/EOS-5DS-R---Lenses-tested

They have published their testing of EF lenses on the 5DS R for the first time. No testing on the 5DS that I can see (yet). But this represents a good 100+ EF lenses retested in a 50 MP context, so love or hate DXO, I think you'll be hard pressed to find another site so comprehensively trying to answer the questions of "How much of the 50 MP will my lens actually 'see'?" or "Is my lens future-proofed for the high MP world?"

None of the rankings or 'lens selector' doo-dads or lens baseball cards report the new scores, but if you choose a lens and select the body it's on as the 5DS R, the new scores are there.

- A

 

[ahsanford]

Some interesting selections:

Canon 35L II
5DS R = 37 P-MPix / 42 Overall Score
5D3 = 18 P-MPix / 33 Overall Score

Sigma 35 Art
5DS R = 34 P-MPix / 41 Overall Score
5D3 = 19 P-MPix / 34 Overall Score

(so apparently, the absurd average-hurting softness of the Canon @ f/22 vs. the Sigma tapping out at f/16, which rated the Art higher than the L, were overcome with pixels this time...)

----------------

Canon 50L
5DS R = 21 P-MPix / 35 Overall Score
5D3 = 16 P-MPix / 29 Overall Score

(very crudely from just the sharpness overall metric, this implies the 50L doesn't benefit from all those pixels as much as sharper lenses might, which should not be surprising)

Zeiss 55 Otus
5DS R = 37 P-MPix / 46 Overall Score
5D3 = 21 P-MPix / 38 Overall Score

Sigma 50 Art
5DS R = 40 P-MPix / 44 Overall Score
5D3 = 21 P-MPix / 35 Overall Score

(first ever report of the Art outresolving the Otus?)

- A

 

[ahsanford]

But DXO is still DXO.

Classic.

 

[bdunbar79]

Boy that 300mm f/2.8L II IS lens sure seems like a truly perfect lens, huh? That is incredible.

 

[ahsanford]

If curious -- DXO's P-MPix sharpness assessment of how much of the 50 MP sensor is realized by these lenses:

Canon 11-24L = 26
Sigma 20mm Art = 40
Canon 16-35 f/4L IS = 23
Canon 24-70 f/2.8L II = 32
Canon 24-70 f/4L IS = 21
Sigma 24 Art = 30
Sigma 24-35 Art = 32
Canon 35 L II = 37
Canon 35 f/2 IS USM = 34
Sigma 35 Art = 34
Canon 50L = 21
Sigma 50 Art = 40
Zeiss 55 Otus = 37
Canon 70-200 f/2.8L IS II = 33
Tamron SP 70-200mm F/2.8 Di VC USD = 36
Canon 85L I = 30
Canon 85L II = 23
Zeiss 85 Otus = 41
Canon 100L = 24 (this may be because it stops down to f/32, which crushes its average)
Canon 100-400L II = 24
Canon 135L = 30
Canon 200-400 no extender = 33
Canon 200-400 with extender = 18
Canon 300 f/2.8L IS II = 45 (highest they've ever reported)
Canon 500L II = 31
Canon 600L II = 37

Shaded in red are where other non-DXO tests, general reputation, etc. would question those numbers. That 16-35 f/4L IS figure, in particular, would go against the grain of virtually every review.

Feel free to mine the original link at the top -- something you may have wanted compared on 50 MP is probably on there.

- A

 

[ahsanford]

And, unless I've read something incorrectly, despite have +14 MP over the D800E, only the Zeiss 135mm on a 5DS R will crack DXO's top five lens + body combos. So slapping the same damn Sigma Art or Zeiss Otus lens on that 5DS R will do you no better than the D800E.

If you're shocked at that, just remember who's reporting the data. :

- A

 

[ahsanford]

Also, another peach from what they released today.

Apparently, everything else being the same -- except the Canon vignetting a shade more yet outresolving the Nikon -- the Nikon gets the higher score.

So, now, after a few years of the same lens being world class or disappointing depending on how many pixels were sitting behind it no longer applies. It's all about the vignetting, people. Brighter corners are the new megapixels.

#dxo #fairandbalanced

- A

 

[Proscribo]

Also, another peach from what they released today.

Apparently, everything else being the same -- except the Canon vignetting a shade more yet outresolving the Nikon -- the Nikon gets the higher score.

So, now, after a few years of the same lens being world class or disappointing depending on how many pixels were sitting behind it no longer applies. It's all about the vignetting, people. Brighter corners are the new megapixels.

#dxo #fairandbalanced

- A

I would say that it's more thanks to the Awesome Dynamic Range provided by those Awesome Sony Sensors!

Not like it would have anything to do with lens performance, but still..

 

[3kramd5]

Also, another peach from what they released today.

Apparently, everything else being the same -- except the Canon vignetting a shade more yet outresolving the Nikon -- the Nikon gets the higher score.

So, now, after a few years of the same lens being world class or disappointing depending on how many pixels were sitting behind it no longer applies. It's all about the vignetting, people. Brighter corners are the new megapixels.

#dxo #fairandbalanced

- A

I would say that it's more thanks to the Awesome Dynamic Range provided by those Awesome Sony Sensors!

Correct, DXO uses sensor DR in their lens scoring algorithm scheme.

Boy that 300mm f/2.8L II IS lens sure seems like a truly perfect lens, huh? That is incredible.

Quite!

 

[ahsanford]

Their databases haven't updated, but you can you select the 5DS R on the lens tested tab.

I can't link it as it's not URL-discrete, but I snagged a few screenshots. The first is sorted on overall score and the second is sorted on sharpness.

(The boxed number under the score/value equals their global ranking at the site across all mounts / designs.)

1) Three Zeiss + one Sigma lens on the 5DS R cracked their top 10 for ratings.

2) Not surprisingly, the 9 sharpest lenses they have tested all came on the 5DS R.

- A

 

[Maximilian]

Once again:

DxO:

Do optical measurments and get data for your SW.
Exit writing reviews and rankings.
Optimize you products.

That will increase your reputation and profit much more than the biased scores you do now.

 

[ahsanford]

Once again:

DxO:

Do optical measurments and get data for your SW.
Exit writing reviews and rankings.
Optimize you products.

That will increase your reputation and profit much more than the biased scores you do now.

Unfortunately, they are the biggest show in town and we may be stuck with them. Consider: PhotoZone and LensTip have tested a combined total of two lenses on the 5DS / 5DS R to date.

- A

 

[chromophore]

Once again:

DxO:

Do optical measurments and get data for your SW.
Exit writing reviews and rankings.
Optimize you products.

That will increase your reputation and profit much more than the biased scores you do now.

Unfortunately, they are the biggest show in town and we may be stuck with them. Consider: PhotoZone and LensTip have tested a combined total of two lenses on the 5DS / 5DS R to date.

- A

DxO only has as much credibility as photographers are willing to give them. Consequently, the innumeracy of the photographic community as a whole, and the way popular online photo sites and blogs (PetaPixel being one I can think of for example) announce DxO results, only reinforce DxO's position as a source for information about how bodies and lenses perform. When is the last time you saw PetaPixel talk about PZ or LensTip or TDP measurements? But DxO crowns a new sensor or lens, it gets front page news despite their methodology being complete BS.

And that is the fault of photographers who don't know basic statistics or who don't care to know. They just want some number they can spout as if it were gospel. That is the fault of people who repost DxO results on photo forums as if they were the least bit meaningful. That is the sort of behavior that gives DxO the exposure and the attention it wants.

Even the criticisms (this one included) only serve to draw more attention to their testing. No news is bad news: like political candidates, controversy is good--it serves to further polarize opinions, and those who disagree or point out even the most elementary methodological flaws, are conveniently branded as biased and their evidence casually dismissed. It is no different than trying to debunk flat earth theorists with science.

Photographers as a whole choose to be ignorant. For all their attention to MTF curves and the minutiae of sensor technology, as a group, they don't really care to think critically. Flame me all you want. Not everyone is like that. But the ones who do care are a small minority. Most photographers have a hard enough time figuring out f-stops, never mind asking actual mathematics like 95% confidence intervals for the mean MTF at 40 lp/mm.

 

[LSXPhotog]

But DXO is still DXO.

Classic.

This is ABSOLUTE and total proof of DxO manipulating overall scores to meet their goals and agenda. Outscored in every category....but it's mounted to a Canon...deduct 10-points.

What a shame. They have within their power to create a fair system but choose to not even hide their bias...the DxO One Camera was hilarious. 'Offers super resolution mode where it combines several images to create a cleaner image...something many other cameras do, but we won't test any of them for that.'

 

[Maximilian]

...
It's all about the vignetting, people. Brighter corners are the new megapixels.
...

Could anyone please tell me how the vignetting of a lens could differ by the type of FF sensor sitting behind it???
That's absolute rubbish. If not corrected by SW vignetting will stay the same for the same optical formula.

5Ds/R sensor size: 36 mm x 24 mm
D810 sensor size: 35.9 mm x 24 mm

Please don't tell me it's about this 0.1 mm. And please don't tell me Zeiss and Sigma have a different optical design for Canon and Nikon.
RUBBISH.

 

[Proscribo]

...
It's all about the vignetting, people. Brighter corners are the new megapixels.
...

Could anyone please tell me how the vignetting of a lens could differ by the type of FF sensor sitting behind it???
That's absolute rubbish. If not corrected by SW vignetting will stay the same for the same optical formula.

5Ds/R sensor size: 36 mm x 24 mm
D810 sensor size: 35.9 mm x 24 mm

Please don't tell me it's about this 0.1 mm. And please don't tell me Zeiss and Sigma have a different optical design for Canon and Nikon.
RUBBISH.

Smaller pixels. I'd think the vignetting here isn't caused by the lens but rather the sensor.

 

[privatebydesign]

...
It's all about the vignetting, people. Brighter corners are the new megapixels.
...

Could anyone please tell me how the vignetting of a lens could differ by the type of FF sensor sitting behind it???
That's absolute rubbish. If not corrected by SW vignetting will stay the same for the same optical formula.

5Ds/R sensor size: 36 mm x 24 mm
D810 sensor size: 35.9 mm x 24 mm

Please don't tell me it's about this 0.1 mm. And please don't tell me Zeiss and Sigma have a different optical design for Canon and Nikon.
RUBBISH.

I was wondering that earlier, I came to the conclusion that it is down to the microlens array and filter stack on top of the sensor. We all agree DxO come up with some very fishy numbers, but their core business is selling lens correction software and I believe those figures.

Now I am not saying that what they say is the whole truth, for instance where is the 'corner' measured from, etc, but I do believe they have to take an accurate measurement of the vignetting to make their lens corrections valuable, and they are very good from what I have seen.

 

[Maximilian]

...
It's all about the vignetting, people. Brighter corners are the new megapixels.
...

Could anyone please tell me how the vignetting of a lens could differ by the type of FF sensor sitting behind it??? ...

Smaller pixels. I'd think the vignetting here isn't caused by the lens but rather the sensor.

I was wondering that earlier, I came to the conclusion that it is down to the microlens array and filter stack on top of the sensor.

Thanks, guys, for helping me with this. I didn't come up with these ideas but I think they're the right explanation.
But still it's interesting that this vignetting seems to have a much higher influence to the scores as the resolution has (now?!).
And still it's difficult for me as technican we're not measuring the lens itself but more or less the sensor behind it *sigh*
No, better say the whole combination. And therefore a solitary DxO scoreboard for lenses over different camera brands or even bodies is not worth a cent.

Conclusion for DxO scores:
You can only compare lenses with the same camera body.
Or you can use it to compare different bodies within a brand with the same lens.
Or you can compare different brands with a third party lens.

Conclusion for myself:
Go shooting! Forget about DxO scores!

 

[justaCanonuser]

What really amazes me is that Canon's EF 1.2/85 II with 38 P-Mpx (whatever P-Mpx is) turns out to be sharper than the classic Zeiss 1.4/85 with 33 P-Mpx. The Otus of course is with 47 P-Mpx much sharper, as expected.

In real life I am always am impressed by the results my EF 85/1.2 can deliver, in particular this special blend of decent in-focus sharpness and creamy bokeh. This is the first time I find my own real life experience reflected in DxO lab data. Canon really did a great job with that monster of a superfast lens!

 

[justaCanonuser]

Conclusion for myself:
Go shooting! Forget about DxO scores!

+1 ... Soon I go shooting and forget about DxO by myself ;D