DxO & MTF Charts ... a little help please!

[Rienzphotoz]

BACKGROUND:
I am not into lens/sensor scores or MTF chart readings etc ... I have nothing against it, in fact I envy those who can understand all the math/science (that goes into the lens/senor scores/charts) and make their purchase decisions based on it ... but I generally do not have patience/skill to go into scores and charts ... I buy lenses based on the reviews from a few sources that I trust (including several members in CR) ... sometimes, I buy camera gear based on how comfortable I am with it ... but generally, I feel that most modern camera gear is more capable then my skills as a photographer.

WHAT I WANT TO LEARN & UNDERSTAND:
I would like to learn & understand how someone gives a score to a particular lens or sensor. In this case I chose DxO because, over the past few years, I've read several discussions about DxO (prior to that I was not even aware such a company exists) and it got me interested in wanting to learn a bit more. Generally discussions related to DxO seem to invite war, blood bath and then end up with personal insults (I am also equally to blame for some of those discussions). So, I'd like to request your assistance to help me (and anyone else who is interested) understand DxO scores in "layman's" terms ... but please, I beg you not to turn this into DxO worshiping / hating contest. Also, please refrain from using this topic to poke fun or insult or score a point over fellow member ... we come here because of our passion for photography, so we are basically like minded citizens ... so please help me (and others) learn.

NOW THE QUESTION:
I was looking at some lens comparison scores over at DxO today (a screen shot of which I've posted below). According to this a $3000 dollar lens seems to have better sharpness than a $12000 lens. I do understand that a lens is not just about sharpness but also about various factors such as how well it auto focuses ... but for now I just want to understand only the sharpness part of the scores. From my understanding, the scores suggest that the Sigma 120-300 f/2.8 is sharper than the EF 200-400 L IS (without using 1.4 TC), when both have them are tested at f/4 ... is that correct?
... also, is there an explanation to DxO scores? as in how they have arrived at that score?
... and if there is no explanation (for how they allotted those scores), how does one know if they are accurate?
Appreciate if you can assist in helping me understand in simple terms without jargon ... and once again, no blood bath please.

Thanks in advance

 

[neuroanatomist]

Re: DxO ... a little help please!

(a screen shot of which I've posted below)

?

But if you intended to post a shot of this, then yes they are showing that the Sigma 120-300/2.8 is slightly sharper than the 200-400L. It's probably a subtle difference - 1 P-Mpix on the 5DIII, but no difference on the 6D for example.

... also, is there an explanation to DxO scores? as in how they have arrived at that score?
... and if there is no explanation (for how they allotted those scores), how does one know if they are accurate?

For the Scores, not really. That's one of the big complaints about DxO - their Scores are a 'black box'. There are some details available. When you look at the table that I presume you meant to post as a screenshot, you would be tempted to think that the Score at the top is derived primarily from the Measurements below…but that's not the case, with the exception of transmission. The Score is based on 'performance in 150 lux illumination at 1/60 s and ISO 100'. That means that 1) a faster lens will score higher (thus, the 50/1.8 scores higher than the 600/4) and 2) the same lens will score higher when tested on a body with more DR/color depth at ISO 100 (which means the Lens Score is as much about the sensor as the lens). The reason the 120-300/2.8 scores higher than the 200-400/4 is really down to f/2.8 vs. f/4.

There are several examples of their measurements being incorrect, some of which they've corrected (silently).

 

[Rienzphotoz]

Re: DxO ... a little help please!

(a screen shot of which I've posted below)
?

Ooops ... sorry, I've uploaded the screen shot now

 

[Rienzphotoz]

Re: DxO ... a little help please!

yes they are showing that the Sigma 120-300/2.8 is slightly sharper than the 200-400L. It's probably a subtle difference - 1 P-Mpix on the 5DIII, but no difference on the 6D for example.
The Score is based on 'performance in 150 lux illumination at 1/60 s and ISO 100'. That means that 1) a faster lens will score higher (thus, the 50/1.8 scores higher than the 600/4) and 2) the same lens will score higher when tested on a body with more DR/color depth at ISO 100 (which means the Lens Score is as much about the sensor as the lens). The reason the 120-300/2.8 scores higher than the 200-400/4 is really down to f/2.8 vs. f/4.

Thanks for the reply ... but I may have not understood correctly, let me explain what I understand from that in my own stupid way:
So, they took a 50 f/1.8 lens and shot an image at f/4 ... in order to compare sharpness they would have to do that with a totally different image made with a 600 f/4 L IS lens correct? (because the FOV would be totally different) so they then compared those 2 different images and found that the image made with 50mm was sharper (from my limited understanding, generally subjects closer are lot more sharper then those at a distance).

Now, if they had compared the same image for the both lenses, then the 50mm lens would obviously lose out as they would need to crop the image to get the same FOV).

But if they compared two different images at 2 different focal lengths (and obviously in two different conditions), how can sharpness be defined in a generalized score? wouldn't that be like scoring a 1st grade student's score (who gets 95 out of 100 in math) with a University student's score (who gets 85 out of 100 in math) and then come up with the result that the 1st grade student is sharper?

 

[neuroanatomist]

Re: DxO ... a little help please!

I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.

 

[Rienzphotoz]

Re: DxO ... a little help please!

I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.

So, do they provide any explanation for that?

 

[neuroanatomist]

Re: DxO ... a little help please!

I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.

So, do they provide any explanation for that?

It's the one I gave above - in a dimly lit warehouse (150 lux) with a camera that has no setting higher than ISO 100, the 50/1.8 would be a better choice than an f/4 lens.

The point is, the DxOMark Score isn't derived from the optical measurements (except transmission).

 

[Rienzphotoz]

Re: DxO ... a little help please!

I'm distinguishing between the DxOMark Score (where the Sigma 120-300/2.8 gets a 29 and the Canon 200-400/4L gets a 24) vs. sharpness (which DxO reports in P-Mpix). On the 5DIII, the 50/1.8 gets a Score of 28, higher than the 200-400/4, although the 50/1.8 only delivers 14 P-Mpix vs. 19.

So, do they provide any explanation for that?

It's the one I gave above - in a dimly lit warehouse (150 lux) with a camera that has no setting higher than ISO 100, the 50/1.8 would be a better choice than an f/4 lens.

The point is, the DxOMark Score isn't derived from the optical measurements (except transmission).

Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
Thanks for your patience

 

[neuroanatomist]

Re: DxO ... a little help please!

Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
Thanks for your patience

Well, I'm sure they don't shoot in a dimly lit warehouse, but they are illuminating the target with 150 lux, and that's aout the level of light in a dim warehouse.

For the Overall Score, they are comparing the lenses at the aperture they determine is best, with is stated for each lens on the comparison pages (although not in the table you posted). That 'best at' value is pretty much always wide open for every lens (which makes sense in 150 lux illumination).

 

[Rienzphotoz]

Re: DxO ... a little help please!

Sorry, I still didn't get it ... when you say they shoot in a dimly lit warehouse, they are using f/1.8 for EF 50mm f/1.8 lens and f/4 for the 600mm f/4 lens? or are they shooting both at f/4?
Thanks for your patience

Well, I'm sure they don't shoot in a dimly lit warehouse, but they are illuminating the target with 150 lux, and that's aout the level of light in a dim warehouse.

For the Overall Score, they are comparing the lenses at the aperture they determine is best, with is stated for each lens on the comparison pages (although not in the table you posted). That 'best at' value is pretty much always wide open for every lens (which makes sense in 150 lux illumination).

Hmmm :-\ ... I don't mean to take sides, but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.

 

[neuroanatomist]

Re: DxO ... a little help please!

...but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.

Exactly.

 

[mackguyver]

Re: DxO ... a little help please!

The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view. For instance, I compared the following aspects of lenses (see screen shots below):

1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness?
DxOMark shows the 300mm is sharper across the entire frame.

2. For architectural work, how much worse is the distortion on my 24-70 f/2.8 II than my 24mm 1.4 II?
DxOMark shows the 1.4 II has better control of distortion.

3. How does much better is chromatic aberration on the Zeiss Otus 55mm vs. my 50 f/1.2 when shot wide open? This is something I'm not fond of on , but I wondered, is it that much better to justify the cost?
DxOMark show almost no CA on the Zeiss, while the Canon, well...

Note: I just noticed that I forgot to change the camera body to 5DIII on all of them, which is important for sharpness measurements, but not distortion and others as long as you match FF to FF or crop to crop.

 

[Rienzphotoz]

Re: DxO ... a little help please!

...but such a test does not seem fair or reasonable for a tool (lens) that is made to be used in diverse conditions.

Exactly.

Thanks for your explanations Neuro ... all this got me thinking that for a company like DxO that is capable/equipped to conduct sophisticated lens/sensor tests, what stops them from comparing lenses/sensors at 10 different ISO measurements and in 10 different conditions with 10 different subject movements and then come up with scores in a chart, so people can come to a more informed conclusion :-\

For example they could, in controlled environment, throw a ball and see how well the camera/lens can auto focus at various apertures and then come up with several scores for sharpness ... I think such scores would help customers to figure out what would be a good lens for them for a particular shooting style e.g. if I like to shoot birds in flight in a certain lighting condition, I would have a ready reference of tests/scores to see what fits my needs. Coz they seem to do that very well for images shot at ISO 100 and pretty much everyone seems to agree with DxO that the Sony/Nikon 36MP sensor delivers great results at that ISO ... surely a company that boasts of sophisticated lab tests, is capable of conducting varied tests and producing different scores for different situations. Would that be an unreasonable expectation?

 

[neuroanatomist]

Re: DxO ... a little help please!

The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view.

Just watch out for their mistakes. For example, compare the 17-40L wide open to the 16-35L II stopped down two stops to f/5.6, and check out the field maps. The 17-40 is infamous for mushy corners wide open, but you don't see that on DxO. Is it believable that the corners of the 17-40 wide open are sharper than the corners of the 16-35 at f/5.6? Not to me…

 

[Rienzphotoz]

Re: DxO ... a little help please!

1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness?
DxOMark shows the 300mm is sharper across the entire frame.

In the first set of images (150-600 vs 300), the 300mm is evenly colored, while the 150-600 isn't ... is that representing sharpness or CA?

 

[mackguyver]

Re: DxO ... a little help please!

The best way to use DxO's measurements are for comparison in scenarios you would actually use. I particularly like to use the the "Field Map" view.

Just watch out for their mistakes. For example, compare the 17-40L wide open to the 16-35L II stopped down two stops to f/5.6, and check out the field maps. The 17-40 is infamous for mushy corners wide open, but you don't see that on DxO. Is it believable that the corners of the 17-40 wide open are sharper than the corners of the 16-35 at f/5.6? Not to me…

I'm still scratching my head on those measurements, too, at least after getting over the 70-200 f/2.8 IS I/II review

I always like to use multiple resources - Lenstip, SLRgear, & Photozone are usually the three that seem to have the best reviews/measurements. As they're all free, it makes it pretty easy to be an informed consumer now that we've reached the death of almost all photo shops. I like to borrow lenses from CPS, too, before making big purchases.

1. How does the new Tamron 150-600 compare to my 300 f/.28 II IS at 300mm & f/5.6 in terms of sharpness?
DxOMark shows the 300mm is sharper across the entire frame.

In the first set of images (150-600 vs 300), the 300mm is evenly colored, while the 150-600 isn't ... is that representing sharpness or CA?

That shot represents sharpness.

 

[meli]

Re: DxO ... a little help please!

... all this got me thinking that for a company like DxO that is capable/equipped to conduct sophisticated lens/sensor tests, what stops them from comparing lenses/sensors at 10 different ISO measurements and in 10 different conditions with 10 different subject movements and then come up with scores in a chart, so people can come to a more informed conclusion :-\

...and with 10 different samples of the same camera/lens picked with appropriate criteria.
They probably could, but they wont. What you wish takes time, personnel, and money. And all this to run a free service. That doesnt fly really well.

For example they could, in controlled environment, throw a ball and see how well the camera/lens can auto focus at various apertures and then come up with several scores for sharpness ...

Dxo doesnt do AF. Its a whole other beast and frankly its difficult to come up with an objective/exhaustive suite of lab/field AF tests.

 

[lescrane]

Re: DxO ... a little help please!

So since some of you are willing to explain wtf these charts mean to those of us who have never seen them, can you give a very basic explain. of this chart?
Obviously the darker green the better, the more uniform the better, but what does the square area mean? Is it the frame of the image area? eg, darker in the middle is sharper, lighter around the edges, less sharp?

 

[mackguyver]

Re: DxO ... a little help please!

So since some of you are willing to explain wtf these charts mean to those of us who have never seen them, can you give a very basic explain. of this chart?
Obviously the darker green the better, the more uniform the better, but what does the square area mean? Is it the frame of the image area? eg, darker in the middle is sharper, lighter around the edges, less sharp?

Yes, that it exactly right. The rectangle is the sensor area and darker (green, vs. yellow or red) and more uniform is better

In this case, the Tamron puts in a pretty good showing considering the price difference.

 

[AlanF]

Re: DxO ... a little help please!

Mac
Thanks for bringing up this method of comparison. I have just used it to compare the 150-600mm with the 100-400 in a thread about the Tammy to disprove an assertion that it is a confirmed soft lens.

DxO measurements are very useful for comparing different lenses on the same body or even better the same lens on different bodies.