DXO likes a Canon-made lens for the first time -- mass riots ensue

[ahsanford]

Saw this pic on my RSS feed (see below), but when I went to the link:
http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ#8508

I got a 404 / Page Not Found error. I also couldn't find any 11-24 f/4L testing or reviews on their site through the page for the lens. I wonder if it was taken down due to an editing error or something -- I would check that link again later today.

But it sure looks like it either fared very well in testing or it was tested on a 50 MP body.

Just kidding, the first couldn't have happened without the second -- it's DXO, after all. :

- A

 

[ahsanford]

And there it is on the main page, so no, it's not a fluke. Still a dead link, though.

A 22 score all but certainly means that they are still not ready* for a 50MP test camera. This must be a 1Ds III or 5D3 test.

*Ready, as defined by DXO, is after the date the DXO Lens Score 2.0 can be released. DXO Lens Score 2.0 is defined by the ground-breaking, industry first of not having the sensor resolution affect the score. The timing of DXO Lens Score 2.0 being released before any Canon 5DS-based lens testing can occur is strictly coincidental. But fear not, Canon lovers: the 70-200 f/2.8L IS II remains the 1,064th best lens in our system which we all agree to be true.

- A

 

[ahsanford]

And now the review is up. See the original link.

They also ever-so-kindly did a head to head against the Nikon 14-24, but they thoughtfully only shared D3X + 14-24 data instead of D8x0 + 14-24 data just to keep the resolution of the sensors close.

http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ/Canon-EF-11-24mm-f4L-USM-vs.-Nikon-AF-S-Nikkor-f2.8G-ED-Comparable-performers

- A

 

[Diltiazem]

And now the review is up. See the original link.

They also ever-so-kindly did a head to head against the Nikon 14-24, but they thoughtfully only shared D3X + 14-24 data instead of D8x0 + 14-24 data just to keep the resolution of the sensors close.

http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ/Canon-EF-11-24mm-f4L-USM-vs.-Nikon-AF-S-Nikkor-f2.8G-ED-Comparable-performers

- A

In the comparison and measurement pages the score is 21, not 22. Not sure why.

 

[ahsanford]

And now the review is up. See the original link.

They also ever-so-kindly did a head to head against the Nikon 14-24, but they thoughtfully only shared D3X + 14-24 data instead of D8x0 + 14-24 data just to keep the resolution of the sensors close.

http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ/Canon-EF-11-24mm-f4L-USM-vs.-Nikon-AF-S-Nikkor-f2.8G-ED-Comparable-performers

- A

In the comparison and measurement pages the score is 21, not 22. Not sure why.

Wow. Good catch. Their system (in the simple, single number baseball card score I sent a screen shot of) must only take the highest score a lens gets with any single body.

11-24 f/4L + 1Ds III --> score of 19
11-24 f/4L + 1DX --> score of 22
11-24 f/4L + 5D2 --> score of 20
11-24 f/4L + 5D3 --> score of 21

But that's just a guess. Trying to divine DXO's inner workings is not unlike navigating the Pentagon without a map.

- A

 

[KBStudio]

One problem with DXO lens comparison between manufacturers is that the camera used skews the rating. If you look at lens results for a single body you can effectively evaluate the lens in comparison to all other lenses tested on that body. You can not compare Canon to Nikon to Sony to ... I have always been surprised that these lens reviews do not use an optical bench with the Air Force's glass resolution chart. I was lucky enough to have a camera repair shop that did have just this. We would take all my new lenses and evaluate them before I accepted them. We able to determine the best aperture for the sharpest image and how focus held up on the edges. Anyway, this technique was and is camera independent and the only way to evaluate across manufactures.

 

[Maiaibing]

One problem with DXO lens comparison between manufacturers is that the camera used skews the rating. ...Anyway, this technique was and is camera independent and the only way to evaluate across manufactures.

Not to hijack this thread I will start by saying I've been very happy with my 50mm ART. I could never get myself to buy the 50L even if I have used it much. I never touched it after getting the 50mm ART. Bokeh wise the samples are convincing. However, in my own use I sometimes see some of the hash lines that the ART has been criticized for. Like the 70-200 f/2.8 IS L II vs the old model I think that sometimes the softer lens wins in certain bokeh conditions exactly because they are softer wide open.

On DxO you can even stronger argue that lens test without a body is meaningless because the lens is either on a body or its not taking any pictures.

What DxO has made clear to all of us is that the same lens will perform very differently from one body to the next and looking at the lens/body combo may be quite important even when you want to choose your body. I for one am therefore very happy with DxO as an informative resource.

DxO's problem is that people are unaware/ignorant of their testing methodology and jump to all kinds of conclusions based on their numbers. Personally, I know of no other test site that provides more info on their testing methodology than DxO does. Read it and use the results accordingly. There is a reason other review sites are linking up with DxO.

 

[Loswr]

DxO's problem is that people are unaware/ignorant of their testing methodology and jump to all kinds of conclusions based on their numbers.

DxO's problems are 1) that their Lens Score is based primarily on the Sensor Score of the camera and only one of the optical metrics they report (transmission), and 2) that they make egregious errors in some cases, and support their erroneous data (unless they silently correct it as they have sometimes done).

 

[jeffa4444]

Test the lens (MTF and projection plus f stop measurement), test the camera with a "lab lens"(this lens usually 50mm has known abberations which can be eliminated from any calculation) (we can test actual DR and the variation from one camera to another of the same type) then test the lens on the camera under controlled lighting, if they have video we can project images onto a 50ft screen noting fall off, resolution, distortions, abberations in much the same way as the lens on a projector these principles can be employed for both stills and video.

 

[LetTheRightLensIn]

And now the review is up. See the original link.

They also ever-so-kindly did a head to head against the Nikon 14-24, but they thoughtfully only shared D3X + 14-24 data instead of D8x0 + 14-24 data just to keep the resolution of the sensors close.

http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ/Canon-EF-11-24mm-f4L-USM-vs.-Nikon-AF-S-Nikkor-f2.8G-ED-Comparable-performers

- A

In the comparison and measurement pages the score is 21, not 22. Not sure why.

Wow. Good catch. Their system (in the simple, single number baseball card score I sent a screen shot of) must only take the highest score a lens gets with any single body.

11-24 f/4L + 1Ds III --> score of 19
11-24 f/4L + 1DX --> score of 22
11-24 f/4L + 5D2 --> score of 20
11-24 f/4L + 5D3 --> score of 21

But that's just a guess. Trying to divine DXO's inner workings is not unlike navigating the Pentagon without a map.

- A

the overall lens score ratings at DxO really make no sense
they will compare zooms against one another or with each set to a different aperture and focal length!
and they are obsessed with T-stops and wide open shooting

it's tricky enough to try to boil a sensor down to one number and kinda of silly, but with lenses it's just beyond silly

 

[fragilesi]

the overall lens score ratings at DxO really make no sense
they will compare zooms against one another or with each set to a different aperture and focal length!
and they are obsessed with T-stops and wide open shooting

it's tricky enough to try to boil a sensor down to one number and kinda of silly, but with lenses it's just beyond silly

Agreed. It's easy to see the attraction of a simple, single score system but unfortunately other than when it comes at the very high or low end (indicating a total stinker or an overall stellar performer) it's not that helpful. In their case I'm not sure it's helpful even on those cases . . .

 

[rfdesigner]

And now the review is up. See the original link.

They also ever-so-kindly did a head to head against the Nikon 14-24, but they thoughtfully only shared D3X + 14-24 data instead of D8x0 + 14-24 data just to keep the resolution of the sensors close.

http://www.dxomark.com/Reviews/Canon-EF-11-24mm-F4L-USM-lens-review-Ultra-wide-champ/Canon-EF-11-24mm-f4L-USM-vs.-Nikon-AF-S-Nikkor-f2.8G-ED-Comparable-performers

- A

In the comparison and measurement pages the score is 21, not 22. Not sure why.

Wow. Good catch. Their system (in the simple, single number baseball card score I sent a screen shot of) must only take the highest score a lens gets with any single body.

11-24 f/4L + 1Ds III --> score of 19
11-24 f/4L + 1DX --> score of 22
11-24 f/4L + 5D2 --> score of 20
11-24 f/4L + 5D3 --> score of 21

But that's just a guess. Trying to divine DXO's inner workings is not unlike navigating the Pentagon without a map.

- A

the overall lens score ratings at DxO really make no sense
they will compare zooms against one another or with each set to a different aperture and focal length!
and they are obsessed with T-stops and wide open shooting

it's tricky enough to try to boil a sensor down to one number and kinda of silly, but with lenses it's just beyond silly

to my eyes they seem more like a landscape obsessed setup.

DR and max sharpness at any aperture seem their two most important metrics.

To me sharpness wide open, and again at something like f2.8 and quality of boke are more important. Almost any lens is sharp at f8. Also colour accuracy is key, something canon has always been good at.. but that means having an overlapping green-red performance which adds noise, DxO penalise canon for that, but photographers like it as it makes for better skin tones because it mimics better what happens in the eye (well that's how I understand it.. I also own astro RGB filters, terrestrial pics taken with those just look wrong as they have almost no overlap)

The big hitters for sports are AF accuracy on moving targets, frame rate and the number of frames you can take at high speed before the buffer's full. They don't test any of these, but the claim high ISO noise = sports.. yeh right!.. even then I'm not sure they give reasonable ISO numbers, once you dig through the data it quickly becomes apparent that DR at high iso isn't the metric they're using to calculate the ISO figure as the 6D can outperform some Nikons at ISO 1600-3200 ish region but gets an ISO rating of around 2400 which is lower than the Nikons.

No.. the headline figures are not worth anything.

The data they publish is worth looking at, just not their take on that data.

 

[zim]

A 22 score all but certainly means that they are still not ready to release the data* for a 50MP test camera.

Fixed that for you ;D

DxO Optics Pro has had that lens and body combo profiled for some time, bottom of this link
http://www.dxo.com/us/photography/photo-software/dxo-opticspro/supported-cameras
They clearly have the data and are choosing not to publish.
On the + side they have published for my old 500D pheew what a relief :