At long last, DXO aggregates all lens testing together

[ahsanford]

Weekes after DXO reported the 5DS R lens testing, the lens test results are finally now in 'gen pop' with the rest of their lens testing in their tables and rankings:

http://www.dxomark.com/lenses#hideAdvancedOptions=false&viewMode=list&yDataType=rankDxo

Joyous DXO magic of note:

• The sharpest 9 lens + body combos in their database are on the 5DS R, yet only 4 of those cracked the DXO top 10 in their absurd lens ranking system.
  
  
• In the following instances of a 'near identical' lens from Zeiss / Sigma being offered on both the EF or Nikon mount, we have the following wonderful findings:
  
  Zeiss Otus 55mm
  On Nikon (D800E): Sharpness = 33 P-MPix, Overall Score = 50
  On Canon (5DS R): Sharpness = 37 P-MPix, Overall Score = 46
  
  Zeiss Otus 85mm
  On Nikon (D800E): Sharpness = 35 P-MPix, Overall Score = 49
  On Canon (5DS R): Sharpness = 41 P-MPix, Overall Score = 47
  
  Sigma 35mm Art
  On Nikon (D800E): Sharpness = 30 P-MPix, Overall Score = 43
  On Canon (5DS R): Sharpness = 34 P-MPix, Overall Score = 41
  
  So if the lenses are effectively identical, one would look at the sensor/mount as a root cause for a lower score. I'm sure someone more versant with DXO's pentagon-like level of transparency can explain how sensor-stack-y transmission differences might explain that scoring discrepancy. Or it's just DXO being DXO.
  
  In fairness, there were some less high profile Sigma / Zeiss lenses that the Canon so comprehensively outresolved the Nikon that even DXO had to give Canon a higher score.
  
  

- A

 

[Loswr]

The Lens Score is based on shooting at base ISO in 150 lux, and clearly a SoNikon sensor means the lens is better at shooting in a dimly lit warehouse.

 

[Don Haines]

The Lens Score is based on shooting at base ISO in 150 lux, and clearly a SoNikon sensor means the lens is better at shooting in a dimly lit warehouse.

For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

 

[rfdesigner]

They really do need to publish their formula, it's getting them so much ridicule.

I have the 100F2.0 myself, chosen ahead of the 85f1.8 partly due to being sharper wide open, the 85 is known for being just a touch soft wide open. But DxO rates the 85 as sharper and then goes on to score the 100 as better overall...

http://www.dxomark.com/Lenses/Compare/Side-by-side/Canon-EF-100mm-F2-USM-on-Canon-EOS-5DS-R-versus-Canon-EF85mm-f-1.8-USM-on-Canon-EOS-5DS-R__798_1009_241_1009

Until they publish their magic scoring formula I'll suspect the defining factor for any lens score is how much they had to drink at lunch.

 

[rfdesigner]

The Lens Score is based on shooting at base ISO in 150 lux, and clearly a SoNikon sensor means the lens is better at shooting in a dimly lit warehouse.

..
Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters....

Or living in the UK... I did see the sun a few weeks ago mind.

 

[ajfotofilmagem]

It is very simple:

A certain lens when mounted on a body with wide dynamic range, magically becomes a better lens.

 

[zim]

Maybe Sony sensors are flater

 

[StudentOfLight]

Intriguing that the Zeiss 135mm f/2 is the highest scoring Zeiss lens on the 5DsR but on the D800E it is only the third highest Zeiss lens. I would have expected the relative positions to remain the same when moving from one system to another.

 

[RGF]

They really do need to publish their formula, it's getting them so much ridicule.

Why? That would be no fun. Let's consider their test to be unbias and above question.

After all, it is on the web, so it MUST be true ;D

 

[ahsanford]

Intriguing that the Zeiss 135mm f/2 is the highest scoring Zeiss lens on the 5DsR but on the D800E it is only the third highest Zeiss lens. I would have expected the relative positions to remain the same when moving from one system to another.

Yep, and the 35L II loses in sharpness to the Sigma 35 Art on the 5D3 yet wins on the 5DS R.

N = 1 lens on 1 body = a statistical grab bag.

- A

 

[Sporgon]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

Dear Dilbert, low light doesn't equal high ISO. Have a good Christmas, Regards, Sporgon.

 

[Rahul]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

I would assume that for testing sharpness, all testing is done at base ISO setting up the camera on a tripod. No? Or maybe you are right - DxO has lost its marbles and is testing for sharpness shooting handheld :

 

[Loswr]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

Dear Dilbert, low light doesn't equal high ISO. Have a good Christmas, Regards, Sporgon.

Lol. Maybe he can ask Santa for better reading comprehension skills, which might help him seem less foolish.

The Lens Score is based on shooting at base ISO in 150 lux, and clearly a SoNikon sensor means the lens is better at shooting in a dimly lit warehouse.

A Lens Score that's strongly influenced by DR and color depth at base ISO makes perfect sense...to DxO, and almost no one else. DxO = BS.

 

[StudentOfLight]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

No, Canon is not behind at high ISO. DXO ISO score does not reflect real world performance. Look at the discrepancy between what is seen in RAW Lab images from DP Review vs DXO ISO score. According to DXO the D800E should be 1/3 stop cleaner than the 5Ds R.

 

[Don Haines]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

It's not a question of who works better at what levels, it's about picking a realistic level to use as the standard.

If it were me, I would have picked a level of somewhere around 25,000 Lux...... a condition that we often shoot at. At 150 Lux, most of us are grabbing flashes and turning on the lights.....

 

[Loswr]

At 150 Lux, most of us are grabbing flashes and turning on the lights.....

Especially if for some reason you were forced to set your camera to base ISO and a 1/60 s exposure. But somehow DxO believes those are the most relevant conditions to test lenses.

 

[ahsanford]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

No, Canon is not behind at high ISO. DXO ISO score does not reflect real world performance. Look at the discrepancy between what is seen in RAW Lab images from DP Review vs DXO ISO score. According to DXO the D800E should be 1/3 stop cleaner than the 5Ds R.

It all depends on whether you are talking about noise at a pixel level or at some normalized resolution. 5D3 has less noise than 5DS at a pixel level, but if you downsample the 5DS to 22 MP, the two are quite similar (some argue the 5DS is better in that comparison).

So at a pixel level, the D800E should outperform the 5DS, shouldn't it?

But your DPR crops that were size-normalized for comparison purposes represent a greater downsampling of the 50 MP image than the 36 MP image, which would reduce noise in that comparison. So those RAW clips you posted can be misleading depending on the context.

Carnathan covers this pretty well.

Comparing a 5D3 to a 5DS R (natively):
http://www.the-digital-picture.com/Reviews/Camera-Noise.aspx?Camera=792&Test=0&ISO=1600&CameraComp=979&TestComp=0&ISOComp=1600

Comparing a 5D3 to a 5DS R (downsampled to 22 MP):
http://www.the-digital-picture.com/Reviews/Camera-Noise.aspx?Camera=792&Test=0&ISO=1600&CameraComp=979&TestComp=1&ISOComp=1600

- A

 

[StudentOfLight]

...
For those readers unfamiliar with how bright 150 lux is, think of a very dark overcast rainy day and that's about the brightness DXO tests under.... sunlight has a brightness from about 50,000 lux to a bit over 100,000 lux.

Another way of putting it is to imagine living your life looking through a pair of glasses with ND-9 filters.... Do that and you can see the world like DXO.....

150 lux should be perfect for cameras that have good high ISO - like Canon cameras do! And everyone here knows how important high ISO is because it is so much more important than low ISO, right?

So if DxO use high ISO in low light, just what lots and lots of people shoot with here and say is more important than low ISO, then why isn't Canon on top?

Or maybe what this suggests is that Canon's high ISO isn't as good as Nikon/Sony in low light situations?

No, Canon is not behind at high ISO. DXO ISO score does not reflect real world performance. Look at the discrepancy between what is seen in RAW Lab images from DP Review vs DXO ISO score. According to DXO the D800E should be 1/3 stop cleaner than the 5Ds R.

It all depends on whether you are talking about noise at a pixel level or at some normalized resolution. 5D3 has less noise than 5DS at a pixel level, but if you downsample the 5DS to 22 MP, the two are quite similar (some argue the 5DS is better in that comparison).

So at a pixel level, the D800E should outperform the 5DS, shouldn't it?

But your DPR crops that were size-normalized for comparison purposes represent a greater downsampling of the 50 MP image than the 36 MP image, which would reduce noise in that comparison. So those RAW clips you posted can be misleading depending on the context.

Carnathan covers this pretty well.

Comparing a 5D3 to a 5DS R (natively):
http://www.the-digital-picture.com/Reviews/Camera-Noise.aspx?Camera=792&Test=0&ISO=1600&CameraComp=979&TestComp=0&ISOComp=1600

Comparing a 5D3 to a 5DS R (downsampled to 22 MP):
http://www.the-digital-picture.com/Reviews/Camera-Noise.aspx?Camera=792&Test=0&ISO=1600&CameraComp=979&TestComp=1&ISOComp=1600

- A

DXO score is based on a same-size scaling, so DPReview's "equalized sizes comparison" is valid image comparison to DXO, as in both cases the images are scaled to the same relative sizes.

 

[StudentOfLight]

...
No, Canon is not behind at high ISO. DXO ISO score does not reflect real world performance. Look at the discrepancy between what is seen in RAW Lab images from DP Review vs DXO ISO score. According to DXO the D800E should be 1/3 stop cleaner than the 5Ds R.

And the pictures you post support that - the 5DSR looks awful compared to the D800E and thus why the D800E would outscore the 5DSR in lens scores.

What about the pictures that I posted support what?
What are you seeing that I am not seeing?

 

[Loswr]

...
No, Canon is not behind at high ISO. DXO ISO score does not reflect real world performance. Look at the discrepancy between what is seen in RAW Lab images from DP Review vs DXO ISO score. According to DXO the D800E should be 1/3 stop cleaner than the 5Ds R.

And the pictures you post support that - the 5DSR looks awful compared to the D800E and thus why the D800E would outscore the 5DSR in lens scores.

What about the pictures that I posted support what?
What are you seeing that I am not seeing?

Someone looking through dilbert-goggles could be seeing anything from noise where there is none to lenses that are cameras to pink elephants flying around.

Friends don't let friends dilbert-goggle!