DXO finally re-tests EF lenses on the 5DS R

[BRunner]

Well, DXO as most of other sites gives us bunch of numbers, how all lenses perform at near field distance at flat target... Don't know how often you do art reproduction or shot brick walls...
I'm interested in performance at macro distances with macro lenses, in infinity performance with almost all lenses, at longer distances with tele-lenses and so on...

Nice example is Zeiss Distagon 2.8/25 ZF, which is optimized for infinity performance at f8-f11. At this settings is one of the best lenses in this FL, yet in all tests it's flop. On the other side, new Zeiss Distagon 2/25 Z* is fantastic at close distances, superior in all tests, but at infinity it never reaches corner performance of it's slower sibling... And DXO can describe all this nuances with ONE NUMBER!

 

[StudentOfLight]

The difference in identical-lens performance on camera A and camera B is predominantly due to ISO performance. A 1stop difference equates to 15pt.
pt = Log2((ISO_A)/(ISO_B)) x(15)

Substituting the ISO scores for D800E (2979) & 5Ds-R (2308)
pt = Log2((2979)/(2308)) x(15)
= 5.52

So identical optical formulas on D800E start with +5.52 more than the 5Ds-R due to noise.

 

[jd7]

DxO only has as much credibility as photographers are willing to give them. Consequently, the innumeracy of the photographic community as a whole, and the way popular online photo sites and blogs (PetaPixel being one I can think of for example) announce DxO results, only reinforce DxO's position as a source for information about how bodies and lenses perform. When is the last time you saw PetaPixel talk about PZ or LensTip or TDP measurements? But DxO crowns a new sensor or lens, it gets front page news despite their methodology being complete BS.

And that is the fault of photographers who don't know basic statistics or who don't care to know. They just want some number they can spout as if it were gospel. That is the fault of people who repost DxO results on photo forums as if they were the least bit meaningful. That is the sort of behavior that gives DxO the exposure and the attention it wants.

Even the criticisms (this one included) only serve to draw more attention to their testing. No news is bad news: like political candidates, controversy is good--it serves to further polarize opinions, and those who disagree or point out even the most elementary methodological flaws, are conveniently branded as biased and their evidence casually dismissed. It is no different than trying to debunk flat earth theorists with science.

Photographers as a whole choose to be ignorant. For all their attention to MTF curves and the minutiae of sensor technology, as a group, they don't really care to think critically. Flame me all you want. Not everyone is like that. But the ones who do care are a small minority. Most photographers have a hard enough time figuring out f-stops, never mind asking actual mathematics like 95% confidence intervals for the mean MTF at 40 lp/mm.

I'm not going to flame you - I'm going to agree with you. And sadly, photographers are far from the only group which can be accused of not thinking critically.

 

[StudentOfLight]

DxO only has as much credibility as photographers are willing to give them. Consequently, the innumeracy of the photographic community as a whole, and the way popular online photo sites and blogs (PetaPixel being one I can think of for example) announce DxO results, only reinforce DxO's position as a source for information about how bodies and lenses perform. When is the last time you saw PetaPixel talk about PZ or LensTip or TDP measurements? But DxO crowns a new sensor or lens, it gets front page news despite their methodology being complete BS.

And that is the fault of photographers who don't know basic statistics or who don't care to know. They just want some number they can spout as if it were gospel. That is the fault of people who repost DxO results on photo forums as if they were the least bit meaningful. That is the sort of behavior that gives DxO the exposure and the attention it wants.

Even the criticisms (this one included) only serve to draw more attention to their testing. No news is bad news: like political candidates, controversy is good--it serves to further polarize opinions, and those who disagree or point out even the most elementary methodological flaws, are conveniently branded as biased and their evidence casually dismissed. It is no different than trying to debunk flat earth theorists with science.

Photographers as a whole choose to be ignorant. For all their attention to MTF curves and the minutiae of sensor technology, as a group, they don't really care to think critically. Flame me all you want. Not everyone is like that. But the ones who do care are a small minority. Most photographers have a hard enough time figuring out f-stops, never mind asking actual mathematics like 95% confidence intervals for the mean MTF at 40 lp/mm.

I'm not going to flame you - I'm going to agree with you. And sadly, photographers are far from the only group which can be accused of not thinking critically.

+2

The fundamental assumption of shooting with 150lux and the subsequent weighting of T-Stop and vignette leads to nonsensical results like this:
http://www.dxomark.com/Lenses/Compare/Side-by-side/Zeiss-Carl-Zeiss-Apo-Planar-T-Star-Otus-85mm-F14-ZF2-Nikon-on-Nikon-D700-versus-Canon-EF-500mm-F4L-IS-II-USM-on-Canon-EOS-5D__1384_441_393_176

and this:
http://www.dxomark.com/Lenses/Compare/Side-by-side/EF50mm-f-1-8-II-on-Canon-EOS-5DS-R-versus-Canon-EF-600mm-F4L-IS-II-USM-on-Canon-EOS-5DS-R__187_1009_394_1009

etc...

 

[3kramd5]

The fundamental assumption of shooting with 150lux and the subsequent weighting of T-Stop and vignette leads to nonsensical results like this:
http://www.dxomark.com/Lenses/Compare/Side-by-side/Zeiss-Carl-Zeiss-Apo-Planar-T-Star-Otus-85mm-F14-ZF2-Nikon-on-Nikon-D700-versus-Canon-EF-500mm-F4L-IS-II-USM-on-Canon-EOS-5D__1384_441_393_176

and this:
http://www.dxomark.com/Lenses/Compare/Side-by-side/EF50mm-f-1-8-II-on-Canon-EOS-5DS-R-versus-Canon-EF-600mm-F4L-IS-II-USM-on-Canon-EOS-5DS-R__187_1009_394_1009

The Otus 85 and the EF500 II are as similar as spectacles and binoculars. It not just that the weighting leads to nonsensical results, the decision to derive a score with no consideration of purpose is nonsense in and of itself.

Shrug. I agree with PBD: DxO's bread and butter is (or at least was in the past, now that they're a camera manufacturer their focus may shift, pun intended) lens measurements and software-side corrections thereof. The aggregate scores exist to drive web traffic, but the underlying data are pretty good.

 

[StudentOfLight]

...

The fundamental assumption of shooting with 150lux and the subsequent weighting of T-Stop and vignette leads to nonsensical results like this:
...

And you know the problem is shooting with 150lux because:
a) you've run DxO's tests with more or less light and got results that you like better
b) someone else said this is a problem so you're repeating that
c) you've read expert commentary from someone else that has tested with more/less light and shown why 150lux is wrong.

If your answer is either (a) or (c), please provide more details.

I choose option d) Dig, investigate and use my brain because I don't need an expert to tell me what to think.

Here are more details: http://www.dxomark.com/About/Lens-scores

DxOMark Score is based on low-light conditions (150 lux and 1/60s exposure time). We chose these conditions because we believe low-light performance is very important in photography today, and because photographers need to know how well lenses perform at their widest aperture. Lenses with a high f-number are usually more expensive, so photographers want to know if the performance is worth the expense. The score does not account for depth of field, and only considers performance at best focus.

e.g. Given 150 lux and 1/60s,
- Lens A with T5.6 needs to shoot ISO 3200 on camera X for a proper exposure.
- Lens B with T1.4 needs to shoot ISO 100 on camera Y for a proper exposure.

The difference between AX and BY is highly dependent on ISO performance of Camera-X vs ISO performance of Camera-Y, and not so much about lens performance X vs lens performance Y. I wonder how noticeable the difference between ISO 3200 and ISO 100 is... ???

Here is the classic example a world class f/4 lens vs an ultra-wide aperture lens:
http://www.dxomark.com/Lenses/Compare/Side-by-side/Canon-EF-600mm-F4L-IS-II-USM-on-Canon-EOS-5DS-R-versus-EF50mm-f-1.2L-USM-on-Canon-EOS-5DS-R__394_1009_197_1009
The 600mm f/4 is significantly better in every metric except T-stop. So the only way for it to score lower than the 50/1.2 is if T-stop is valued much more than any of the other metrics. Surely this points out the problem in the most obvious way.

On to the actual value of 150 lux. I can't speak for others there, but I hardly ever shoot in 150 lux light levels. If I do, it wouldn't be by choice, it would usually be the very the last resort.

FYI for more info on lux: https://en.wikipedia.org/wiki/Lux (150 lux = 1/60s at T1 at ISO 100. This is something like a dimly lit church.)

Is the typical photographer always/predominantly interested in shooting in 150 lux? Also take into account the fact that depth-of-field for different use cases will require you stopping down and for sports you might want to use faster shutters speeds to stop motion. Again, I cannot speak for others out there but I'll answer for the following use cases given 150 lux:
- Portrait Photography? (e.g. T5.6, 1/60s therefore ISO 3200) - No, I add light to shoot at low ISO
- Landscape Photography? (e.g. T8, 1/60s therefore ISO 6400) - No (refer to table in Wikipedia article)
- Commercial photography? (e.g. T5.6, 1/60s therefore ISO 3200) - No, use good lighting and ISO
- Wildlife Photography? (e.g. T5.6, 1/2000s therefore ISO 51200) - No, I call it a day when the sun sets.
- Events photography? (e.g. T4, 1/250s therefore ISO 6400) - Preferably not, I add light when possible
- Wedding photography? (e.g. T4, 1/250s therefore ISO 6400) - Preferably not, I add light when possible
- Sports photography? (e.g. T4, 1/2000s therefore ISO 25600) - Preferably not, but sometimes it's unavoidable
- Astro-photography? (e.g. T2.8, 1/60s therefore ISO 800) Yes, this is the one case where I definitely shoot with much lower light levels.

Do those ISO values reflect real world shooting?

Whenever possible, would a professional not negotiate with relevant stakeholders to organize a better lighting setup or the ability to use flash for the client's images if there was 150 lux ambient? Forget about pro's for a second, green-box shooters will also have the pop-up flash activating automatically when the light levels drop low, so yet again the 150 lux assumption goes out the proverbial window as the pop-up flash will illuminate the subject instead of relying on low ambient light.

Do you really still think 150 lux is a reasonable assumption to apply for overall lens ranking? Surely it is a very specialize use case which does not apply to typical shooting scenarios. Also sports photography, being a possible 150 lux use case would not automatically lean towards an Otus 85mm f/1.4 (manual focus prime lens) for their low light conditions simply due to it's fast T-stop. The Canon EF 200-400mm f/4 L IS USM would be much more appropriate and useful due to reach, autofocus, zooming (to optimize framing) and image stabilization.

Surely lenses should be ranked by use case, and not by one overall number which is heavily influenced by T-stop due to an arbitrary 150 lux constraint.

TL/DR: In summary I will be so audacious as to quote myself:

The fundamental assumption of shooting with 150lux and the subsequent weighting of T-Stop and vignette leads to nonsensical results...

 

[ahsanford]

I personally think 90% of DXO haters will evaporate simply by dropping the aggregate lens score and the cross-platform lens + body rankings. Just report your test data and walk away, i.e. limit yourself to just being a resource and a not a source of opinion.

- A

 

[privatebydesign]

I personally think 90% of DXO haters will evaporate simply by dropping the aggregate lens score and the cross-platform lens + body rankings. Just report your test data and walk away, i.e. limit yourself to just being a resource and a not a source of opinion.

- A

And that won't happen.

The ridicule is a good source of clicks and links and page hits. We are in the age where the Kardashians can become multi millionaires on the back of a 'leaked' porn movie, and DxO can keep their name in the photography headlines by publishing biased sudo science bull.

 

[IglooEater]

...
It's all about the vignetting, people. Brighter corners are the new megapixels.
...

Could anyone please tell me how the vignetting of a lens could differ by the type of FF sensor sitting behind it???
That's absolute rubbish. If not corrected by SW vignetting will stay the same for the same optical formula.

5Ds/R sensor size: 36 mm x 24 mm
D810 sensor size: 35.9 mm x 24 mm

Please don't tell me it's about this 0.1 mm. And please don't tell me Zeiss and Sigma have a different optical design for Canon and Nikon.
RUBBISH.

Coming from DXO causes me to simply assume it's wrong. However, I could believe it from almost anyone else. I'm guessing that more pixels and/or less effective microlenses, and/or deeper pixel wells, and/or something else, could have an effect on the amount of light *measured* at a high angle of incidence - i.e. the corners- thus causing more apparent vignetting on one sensor than on another.

 

[candc]

The thing I am most I am most interested in on the dxo site is the p-mpix measurement but I am not sure how exactly they measure it? This is what I can find on their website

"DxOMark’s new Perceptual MPix measurements are based on acutance and human contrast sensitivity function (CSF) published in recently-released image quality standards from the International Standards Organization (ISO) and the International Imaging Industry Association (I3A). A member of the working groups involved in image quality, DxO Labs has been working diligently with giants in the digital imaging industry such as AMD, Nokia, Kodak, Nvidia, Fujifilm, HP, RIM, Intel, Microsoft, Google, and others."

an example that brings this into question for me is with the 400doii. according to dxo it has 29 p-mpix and the 300ii has 45 on the 5dsr.

According to lensrentals who test the lens on an optical bench the 400 is sharper. 1490 vs 1395

I know from experience that the 400 is super sharp but the contrast is a bit low. generally it needs some boosting in post but the resolution is there. Maybe that has something to do with the much lower rating dxo gives it?

 

[9VIII]

The thing I am most I am most interested in on the dxo site is the p-mpix measurement but I am not sure how exactly they measure it? This is what I can find on their website

"DxOMark’s new Perceptual MPix measurements are based on acutance and human contrast sensitivity function (CSF) published in recently-released image quality standards from the International Standards Organization (ISO) and the International Imaging Industry Association (I3A). A member of the working groups involved in image quality, DxO Labs has been working diligently with giants in the digital imaging industry such as AMD, Nokia, Kodak, Nvidia, Fujifilm, HP, RIM, Intel, Microsoft, Google, and others."

an example that brings this into question for me is with the 400doii. according to dxo it has 29 p-mpix and the 300ii has 45 on the 5dsr.

According to lensrentals who test the lens on an optical bench the 400 is sharper. 1490 vs 1395

I know from experience that the 400 is super sharp but the contrast is a bit low. generally it needs some boosting in post but the resolution is there. Maybe that has something to do with the much lower rating dxo gives it?

As has been said many times, only DXO can say how they come up with their numbers.

http://www.lensrentals.com/blog/2015/01/more-canon-400m-do-ii-comparisons
The lensrentals article was for a single copy, not an average.
Comparing those numbers with TDP suggests the average would be somewhere lower than what Lensrentals had at the time.

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=739&Sample=0&FLI=0&API=0&LensComp=962&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=0


Another example, apparently the 300f2.8ISII has massive differences in resolution depending on orientation, but it's beyond me to see it on the test chart.
http://www.the-digital-picture.com/Reviews/MTF.aspx?Lens=739&FLI=0&API=0&LensComp=705&FLIComp=0&APIComp=0&CT=AVG

 

[YellowJersey]

If you are going to go by dxo scores, you should probably look the individual scores for each category rather than the overall score. If dxo places a huge weight on in vignetting and you don't care about that vignetting, then the overall score is meaningless.

I always take dxo scores with a grain of salt. Their weighting system seems weird to me, as vignetting seems to be the most important factor in determining the lens score. I also question some of their methods. Whether dxo is biased against Canon, I'm not sure. It kind of seems like it, but Canon's sensor tech has been lagging and you don't need dxo to tell you that. I knew something was up when I went shooting with a friend who had a D810 and just comparing the histogram with my 5D mkIII I could tell there was a big DR difference. The lens comparison does raise an eyebrow, though.

But whether dxo is biased, I don't really care. DXO is but one tool and, imo, not the most important or valuable tool out there. I care more about what the reviewers say (Matt Granger, Jared Polin, Tony Northrup, The Camera Store, Dustin Abbott, Photorec Toby, and even those lovable goofs at Digital Rev) than what DXO says. And Canon's a big boy, it doesn't need anyone defending it from the big bad monsters at dxo. Dxo is not personally insulting you for using Canon. So take it on the cheek, recognise Canon's strengths, Canon's weaknesses, and move on.

 

[StudentOfLight]

...
Here are more details: http://www.dxomark.com/About/Lens-scores

DxOMark Score is based on low-light conditions (150 lux and 1/60s exposure time). We chose these conditions because we believe low-light performance is very important in photography today, and because photographers need to know how well lenses perform at their widest aperture. Lenses with a high f-number are usually more expensive, so photographers want to know if the performance is worth the expense. The score does not account for depth of field, and only considers performance at best focus.

..

Well it sounds like DxO have aligned their testing with what commenters on CR think is important: low light photography at wide open apertures.

Wide open apertures is a relative thing. Does a Sony/Canon/Nikon/Samsung/Pentax/Sigma 800mm f/1.2 exist?

Here is another example since you clearly didn't read my well laid out explanation earlier:
http://www.dxomark.com/Lenses/Compare/Side-by-side/Zeiss-Carl-Zeiss-Apo-Planar-T-Star-Otus-85mm-F14-ZF2-Nikon-on-Nikon-D700-versus-Canon-EF-400mm-F4-DO-IS-II-USM-on-Canon-EOS-5D__1384_441_1425_176

Note:

DxOMark Score is a linear scale related to the largest print size that provides excellent quality. Doubling the size of the print requires doubling the DxOMark Score

With a score of 29 vs 17: According to the DXOmark score the D700 with Otus85 images (11MP of perfection) allows you to print 70% larger than what is possible with 5D with 400-DO-II images (12MP of perfection).

Is the claim that Otus 85 (even with a slightly lower resolution sensor) produces 70% larger prints than 400 DO II a fair representation of the relative performance of the two lenses?

 

[3kramd5]

Is the claim that Otus 85 (even with a slightly lower resolution sensor) produces 70% larger prints than 400 DO II a fair representation of the relative performance of the two lenses?

Of course, because you have to downsample (reducing those 12 million perfect pixels) to overcome the 3 stop noise disadvantage of the camera you're mounting the 400 to. Makes perfect sense!

 

[candc]

It seems.that the "p-mpx" number that they give is a conglomerate score of sharpness at different apertures. That may be useful to know but if you want to know sharpness at a specific aperture (wide open comes to mind) then its best to look at the "accutance profile"

The accutance profile for the 400doii and 300ii are very similar wide open but the performance of the 400 drops off much more quickly as its.stopped down.

 

[MrToes]

Some of them scores are quite surprising!?

 

[StudentOfLight]

Is the claim that Otus 85 (even with a slightly lower resolution sensor) produces 70% larger prints than 400 DO II a fair representation of the relative performance of the two lenses?

Of course, because you have to downsample (reducing those 12 million perfect pixels) to overcome the 3 stop noise disadvantage of the camera you're mounting the 400 to. Makes perfect sense!

Exactly, the 150 lux assumption!

 

[rfdesigner]

The madness continues even when you compare canon to canon

http://www.dxomark.com/Lenses/Compare/Side-by-side/Canon-EF-100mm-F2-USM-on-Canon-EOS-5DS-R-versus-Canon-EF-85mm-F12L-USM-on-Canon-EOS-5DS-R-versus-Canon-EF-135mm-F2L-USM-on-Canon-EOS-5DS-R__798_1009_940_1009_630_1009

I'm very happy with my 100 f2.0 but is better than both the 135L and 85L? I don't think so!

There's also so much they don't measure at all, so no one should take any of their ratings as worth more than the pixels they're written with, the underlying data has always seemed fairly genuine though.

 

[JMZawodny]

Boy that 300mm f/2.8L II IS lens sure seems like a truly perfect lens, huh? That is incredible.

Oh this is very bad news. After great effort, I had managed to rationalize not buying the 5DSR and to wait for the 5D4 (or 1DX2). Now this update comes along and it seems I must reconsider my logic. The 300mm f/2.8L II simply requires more pixels. I sort of subconsciously knew this already but have been in denial. I use my 300 a lot and have often wondered why I ran into pixels before I ran out of detail. My only course of action is likely to get 2 new bodies in 2016. Now to construct the argument that I will need to persuade my wife to go along with this.

 

[PhotographyFirst]

The madness continues even when you compare canon to canon

http://www.dxomark.com/Lenses/Compare/Side-by-side/Canon-EF-100mm-F2-USM-on-Canon-EOS-5DS-R-versus-Canon-EF-85mm-F12L-USM-on-Canon-EOS-5DS-R-versus-Canon-EF-135mm-F2L-USM-on-Canon-EOS-5DS-R__798_1009_940_1009_630_1009

I'm very happy with my 100 f2.0 but is better than both the 135L and 85L? I don't think so!

There's also so much they don't measure at all, so no one should take any of their ratings as worth more than the pixels they're written with, the underlying data has always seemed fairly genuine though.

Wait. WTF?

The overall score makes absolutely no sense there. They must have messed up with their calculation or something. How can the 100 f2 be worse or equal in every category, yet be rated higher than the 135 f2?

Man, I feel sort of sorry for any sap who buys lenses based on any DXO score. I guess lazy people get what they deserve sometimes, eh? Still nothing beats just looking at full resolution sample photos of the lens you are interested in.