DXO: Fair and Balanced

[ahsanford]

As part of our recurring 'DXO is Fair and Balanced' investigative journalism series, we offer you the finest insights.

Today's wonderful test of DXO's impartiality is the Sigma 24-35 f/2. It's always great to put effectively the same lens on both a Nikon and Canon body and see what DXO thinks about it.

See how things turned out below. Classic DXO.

- A

 

[Mt Spokane Photography]

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.

 

[ahsanford]

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.

Understood and agree. I just want to see the aggregate lens DxOMark Score die a fiery death.

Let DXO report it's nutty numbers with test shots taken in absurd conditions, state that two very similar lenses have a sharpness difference because one stops down to f/22 while the other only stops down to f/16, etc. but don't dangle such a piñata at us like that overall score. It's misleading at best.

- A

 

[JMZawodny]

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.

Understood and agree. I just want to see the aggregate lens DxOMark Score die a fiery death.

Let DXO report it's nutty numbers with test shots taken in absurd conditions, state that two very similar lenses have a sharpness difference because one stops down to f/22 while the other only stops down to f/16, etc. but don't dangle such a piñata at us like that overall score. It's misleading at best.

- A

The quickest way to bring on a fiery death would be for folks to stop using their website or spreading their opaque results elsewhere around the web.

This is my personal "Don't feed the monsters" philosophy.

 

[Sporgon]

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.

Great fun ! I think ahsandford's point is that percentage-wise (judging by the 'scale') most metrics are either identical or within a gnats whisker of each other, with the expection of sharpness, where the 50 mp Canon has a clearly higher score - yet overall the lens scores lower on the Canon.

So this must suggest that DXO's weighting of vignetting and transmission is huge compared with sharpness.

Personally I think it's hilarious, but the problem is a growing number of other sites on the 'net quote them.

 

[Neutral]

Lens tests cannot be compared across models for obvious reasons. I realize that you were doing this tongue in cheek, but its a good example. The measurements vary from camera to camera because they are not measuring a lens. Some people seem to think they are measuring a lens, but that's not the case.

Great fun ! I think ahsandford's point is that percentage-wise (judging by the 'scale') most metrics are either identical or within a gnats whisker of each other, with the expection of sharpness, where the 50 mp Canon has a clearly higher score - yet overall the lens scores lower on the Canon.

So this must suggest that DXO's weighting of vignetting and transmission is huge compared with sharpness.

Personally I think it's hilarious, but the problem is a growing number of other sites on the 'net quote them.

Look at this from the other angle.
DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.

 

[bdunbar79]

Statements like that confuse me. Who cares if it doesn't resolve as high of a % of MP's off the 5DsR sensor? It still has MORE resolution and MORE detail in the final photo than on the D800. The % or how it gets there are completely irrelevant. So I suppose if I can resolve 100% of the pixels off an 18 MP sensor that is better than 60% of the pixels off a 50 MP sensor? What an asinine comment.

The score is not useful at all. People/consumers see a score. They don't go, "Oh yeah, but wait, that means the camera/lens combo so we should be more careful in evaluating the scores."

This is the problem with DxOmark. And I just don't know why people cannot understand this. I'm convinced that some photographers just simply choose to be ignorant.

 

[ahsanford]

I still see value in DXO's lens testing for one reason: I think having 100 lenses tested on both a 5D3 and a 5DS R is useful.

It lets someone with a 5D3 consider a 50 MP rig (or other higher MP rig) to see how well my current glass might perform on it. I shun the notion that a lens is black/white 'rated for 50 MP' or not -- Roger's early 5D3 vs. 5DS data showed that 4 very different lenses all improved -- but the degree to which they did varied.

I just want to know if a given lens of mine will shine on a higher-resolving sensor, or if I need to budget to replace a few lenses to go along with that shiny new SLR. That is one area DXO can help the world -- crappy metrics and scoring systems be damned. They retest on higher resolving rigs, while other sites like LensTip and PZ are absolutely crawling to upgrade their test rigs and re-test lens.

- A

 

[Loswr]

Look at this from the other angle.
DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

This logic of yours is completely wrong. Let's apply it...

Sigma 50/1.4A:
• Nikon D800E, 23 P-Mpix / 36 MP = 64%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95%
• Clearly on the 5DIII the score should be much higher than the D800E, the lens is far less optimal for the D800E
• DxO Lens Score on D800E = 36, Score on 5DIII = 35

Zeiss 50/1.4 Otus
• Nikon D800E, 33 P-Mpix / 36 MP = 92%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95%
• Clearly on the 5DIII the score should be slightly higher than the D800E, the lens is a bit less optimal for the D800E
• DxO Lens Score on D800E = 50, Score on 5DIII = 38

The same would be true of pretty much every 3rd party lens tested on the D800E and the 5DIII, by your flawed logic they should all score higher on the 5DIII - but they don't.

There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.

Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.

What it is all about is that the DxO Score for lenses is based on performance when shooting at base ISO at 1/60 s in 150 lux – the light level of a dimly-lit warehouse. That means the three factors that primarily drive the Lens Score are lens transmission, sensor DR at base ISO, and sensor color depth at base ISO. That's why the nifty-50/1.8 scores higher than the 600/4 II on the same camera, and that's why the score for the Sigma 24-35/2 is higher on the D800E than on the 5DSR.

Good job swallowing DxO's BS hook, line and sinker. Alternatively, maybe you understand perfectly well how DxO's Lens Scores work, and your 'other angle' was an expression of bias rather than flawed logic.

 

[jthomson]

I still like their software. Their testing aggregate score is a joke and is recognized as such.

 

[Mt Spokane Photography]

Look at this from the other angle.
DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.

Good example of a person confused and not understanding what its all about.

Many of us do not understand, and that's because DXO keeps their grading of a lens score a secret. The only thing we know for sure is that companies that buy their products get a higher score, and DXO made cameras get a higher score. They grading their own camera equal to a DSLR, but keep the method a secret as well.
I keep expecting them to get called out on that one by someone like the FTC that requires claims to be substantiated, and you can't keep the method a secret and still claim yours is better.

 

[Monchoon]

Until a better website comes along that provides the same or similar metrics for lenses and sensors, it is the best we've got to use.

The best to use for what? And why would you want another that does the same or similar? :

 

[Aglet]

Until a better website comes along that provides the same or similar metrics for lenses and sensors, it is the best we've got to use.

try

www.lenstip.com

 

[Aglet]

Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.

Perhaps you can dial the vituperation down a stop or 2?

There are plenty of ways of pointing that out without making yourself look like the south end of a north-facing donkey.

 

[Neutral]

Look at this from the other angle.
DXO gives perceptial resolution of the system containing two components - lens and camera sensor.
This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

This logic of yours is completely wrong. Let's apply it...

Sigma 50/1.4A:
• Nikon D800E, 23 P-Mpix / 36 MP = 64%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95%
• Clearly on the 5DIII the score should be much higher than the D800E, the lens is far less optimal for the D800E
• DxO Lens Score on D800E = 36, Score on 5DIII = 35

Zeiss 50/1.4 Otus
• Nikon D800E, 33 P-Mpix / 36 MP = 92%
• Canon 5DIII, 21 P-Mpix / 22 MP = 95%
• Clearly on the 5DIII the score should be slightly higher than the D800E, the lens is a bit less optimal for the D800E
• DxO Lens Score on D800E = 50, Score on 5DIII = 38

The same would be true of pretty much every 3rd party lens tested on the D800E and the 5DIII, by your flawed logic they should all score higher on the 5DIII - but they don't.

There were number of discussions earlier regarding DXO lenses perceptial resolutions on different bodies and what I see that a lot of people are totally confused and do not undestand well what it is all about.

Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.

What it is all about is that the DxO Score for lenses is based on performance when shooting at base ISO at 1/60 s in 150 lux – the light level of a dimly-lit warehouse. That means the three factors that primarily drive the Lens Score are lens transmission, sensor DR at base ISO, and sensor color depth at base ISO. That's why the nifty-50/1.8 scores higher than the 600/4 II on the same camera, and that's why the score for the Sigma 24-35/2 is higher on the D800E than on the 5DSR.

Good job swallowing DxO's BS hook, line and sinker. Alternatively, maybe you understand perfectly well how DxO's Lens Scores work, and your 'other angle' was an expression of bias rather than flawed logic.

Hahaha, this is very funny when someone is trying like a clown to jump out of his pants to show how bad is someone compared to him ( in logic, knowlege, skills etc.) and doing so intentionally distorts what other person was trying to tell or maybe fails to to grasp main point ).
I frequently read this forum just for fun )

May be here was also my fault that I did not formulate precisely main point and left room to allow some people to play with the words. This is the internet forum and not scientific media.
Next time will be more careful to formulate things which I want to share.

For those who did not understand what I was trying to tell repeate just one more time but a bit differently :
Forget about DXO scores - consider this as some sort index in their database.
Other indexes could be created and used for sorting data in better way.
What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
In other words which combo makes better overall SYSTEM - in particular for system resolution.
This especially important for the latest high resolution high performance sensors and see how well lenses match that sensors performance.
This helps a lot (measurement, not scores) to evalute possible SYSTEM components upgrade path.
No more than that, simple as this.
And as for any tool one need to know what it is for and how to use it in a best way.
All heated discussing about scores bashing DXO is just useless waste of time.

 

[StudentOfLight]

I posted this anomaly on the Third Party forum:

http://www.canonrumors.com/forum/index.php?topic=27611.msg568914#msg568914

 

[Neutral]

Unfortunately, it seems you should include yourself in that population of those people who are totally confused and do not understand well what it is all about.

Perhaps you can dial the vituperation down a stop or 2?

There are plenty of ways of pointing that out without making yourself look like the south end of a north-facing donkey.

Alas, there is some category of living beings who think that if they would try to thrust their sh*t onto someone else this could add respect to them or make them more important within community
This particular one seems to belong to this category.
BUT - one need to remember that trying to thrust his sh*t onto others could have result just opposite to the intended - all sh*t could be repelled back and initiator could end up covered with his own sh*t )
Luckily this category is in extreme minority here, most are normal people who do not need to behave the same way to stress that they are better than others.

 

[Neutral]

I posted this anomaly on the Third Party forum:

http://www.canonrumors.com/forum/index.php?topic=27611.msg568914#msg568914

Tehnical discussion of variances in test measurement and trying to interprete differences and trying to understand what is the cause of them is perfectly good one.
Normally with better sensors results should be better but sometimes there could be something hidden under surface that affects results.
Sometimes yes, there could be errors in measurements or records due to number of reasons.
Most outstanding case was with Canon EF 70-200 f/2.8L IS USM II when DXO reported lower perceptial resolution compared to the previous generation, did not publicully admitted that this was wrong but eventially corrected measurements results.
This could be really dissapointing for some but this is not the end of the world ).

 

[Loswr]

Hahaha, this is very funny when someone is trying like a clown to jump out of his pants to show how bad is someone compared to him ( in logic, knowlege, skills etc.) and doing so intentionally distorts what other person was trying to tell or maybe fails to to grasp main point ).

May be here was also my fault that I did not formulate precisely main point and left room to allow some people to play with the words.

Distort? Not understand? Play with words? Let's review your words:

This is very useful to see overal system performance and how well lens is suited for particular camera/sensor.
So on Nikon D800 this combination allows to get 75% of 36mpx sensor resolution (27/36=0.75) but on Canon 5DSR only 64% of 50mpx resolution (32/50=0.64).
So on 5DSR lens degrades overall resolution by 36% and on Nikon d800 only by 25%.
So clearly on D800 score for this lens should by higher than on 5DS.
In other words this lens is less optimal for Canon 5DS than for Nikon D800.

You stated quite clearly that the lens is less optimal on the D800 than on the 5Ds, and thus it makes sense that the score on the D800 is higher. To what 'score' were you referring? Surely not the P-Mpix measurement, because assuming you have basic counting skills you're aware that the system resolution is higher on the 5Ds. No other measurement has relevance. Therefore, you were justifying the higher DxO Score given to the lens on the D800 based on the differential loss of system resolution on the two cameras.

For those who did not understand what I was trying to tell repeate just one more time but a bit differently :
Forget about DXO scores...

When called on your obvious fallacy, you responded: 'Oh, I am just misunderstood, forget about the scores,' instead of simply acknowledging your error.

What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
In other words which combo makes better overall SYSTEM - in particular for system resolution.

For system resolution, the Sigma 24-35 delivers 18% better performance on the 5Ds than on the D800 (32 vs. 27 P-Mpix). So by your own statements, you are confirming that the lens is better on the 5Ds, yet earlier you concluded the opposite – that the lens should get a higher score on the D800. Perhaps you should retract your earlier conclusion. Or just spew more sh!t, it's up to you.

This helps a lot (measurement, not scores) to evalute possible SYSTEM components upgrade path.
No more than that, simple as this.

Yes, that makes perfect sense. DxO's Scores are useless, their measurements are generally useful (except when they make errors). I've probably stated that more times here than your total number of posts on this forum.

 

[privatebydesign]

Forget about DXO scores - consider this as some sort index in their database.
Other indexes could be created and used for sorting data in better way.
What is important about their lens database that this is just tool which allows to estimate how optimal this or that lens on particular body to get maximum performance of the combo.
In other words which combo makes better overall SYSTEM - in particular for system resolution.
This especially important for the latest high resolution high performance sensors and see how well lenses match that sensors performance.
This helps a lot (measurement, not scores) to evalute possible SYSTEM components upgrade path.
No more than that, simple as this.
And as for any tool one need to know what it is for and how to use it in a best way.
All heated discussing about scores bashing DXO is just useless waste of time.

But without knowing and understanding how the tests are made and the results formulated there is nothing of value to that database.

Do we know if there is a 10% gain applied to results from manufacturers that start with S or N? No we don't! Do we know how relevant the testing protocol is to our own actual shooting situations? In general it is nothing like most peoples. Do we know the weighting that is given to lens characteristics that have little impact in real life with the software we have, things like vignetteing? Again we don't.

DXO is entirely flawed and without transparency the 'results', ratings, and numbers are entirely irrelevant.