5Ds/5DsR & Lens resolution - anomaly (Lensrentals ref.)

[Andyx01]

I want to bring to light an odd anomaly on some initial resolution tests published by lensrentals.

Ref: http://www.lensrentals.com/blog/2015/06/canon-5ds-and-5ds-r-initial-resolution-tests

Specifically the Canon 300 2.8 IS II, and Zeiss 85 1.4 Otus results.

Below are the Center sharpness scores of the two lenses on three different bodies:

_________ 300mm _________ 85mm_________ 85>300 result.
5d-III_____1050___________1300___________23.8%
5Ds_______1375___________1570___________14.2%
5Ds-r______1545___________1660___________7.4%


So here is the anomaly. If the center sharpness of the 300mm lens is within 8% of the 85mm lens on a high resolution sensor - one would think that as the sensor resolution DECREASED - the differences would become smaller, but the opposite is happening.

How is this score being generated?

What are the chances of something happening behind the scenes that is affecting results? e.g. in camera distortion correction.

I would like to see the 5D III - 300mm re-tested with the lens not fully seated, (or pins taped over) to eliminate anything that may be occurring due to the camera body identifying the lens.

Even if the results are the same, it brings in to question why this is happening?

It's like examining the paint on a car with a really good magnifying glass and saying paint A is 7.4% smoother than paint B, then having someone else examine the paint with a lower resolution magnifying glass and saying, no paint A is 23.8% smoother than paint B. If anything the magnifying glass that is of lesser quality, should have a harder time detecting the 7.4% difference resulting in a closer score, not a larger gap.

???

 

[Andyx01]

I just want to add the wonderful tool Bryan has over at the-digital-picture.com for comparing lenses:

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=739&Camera=453&FLI=0&API=0&LensComp=957&CameraComp=453&SampleComp=0&FLIComp=0&APIComp=3

To my eye, the center sharpness between the two (both at f/2.8) is roughly the same in the center (not 23.8% worse on the 300.)

While I did not include the corner results, the 300 did score better in the corners than the 85, and the images hosted by Bryan do appear to back that up.

But again, I want to focus on the center descrepancy, and where that score is coming from. It doesn't look right purely by the numbers, and it doesn't look right on the ISO chart either.

 

[Eldar]

This is not my area of expertise, so I´m sure others can be more scientific in their response. My assumptions would be:

The sharpness from a lens-sensor combination will not be linear when you change the resolution of the sensor. At some point, when you increase sensor resolution, you reach the maximum resolution you can get from a lens. With the 5DSR you are moving up the lesser steep part of the curve and sharpness improvements will be less and less. The sensor is less of a limiting factor and the limitations in the lens more dominant.. On the 5DIII, it is the sensor resolution that is more of a limiting factor and you get full effect of the lens quality.

i would also guess that if you did the same measurement on a lower resolution sensor, like a 5DC, the curves would get closer again, since the lesser quality lenses would have enough resolving power.

Looking forward to the qualified answers

 

[Andyx01]

The sensor is less of a limiting factor and the limitations in the lens more dominant.. On the 5DIII, it is the sensor resolution that is more of a limiting factor and you get full effect of the lens quality.

Exactly! The 5D III sensor isn't nearly as discriminating as the 5Dsr. Thus the differences should be getting bigger. Not smaller.

i.e. if you Used a horrid lens on a full frame sensor that only had 1MP of resolution, then compared it to a near perfect lens, the differences would be limited by 1MP and thus not very big, but on a 50MP sensor, the differences should become huge.

Yet.... The opposite is being reported in these tests.

???

 

[Eldar]

The sensor is less of a limiting factor and the limitations in the lens more dominant.. On the 5DIII, it is the sensor resolution that is more of a limiting factor and you get full effect of the lens quality.

Exactly! The 5D III sensor isn't nearly as discriminating as the 5Dsr. Thus the differences should be getting bigger. Not smaller.

i.e. if you Used a horrid lens on a full frame sensor that only had 1MP of resolution, then compared it to a near perfect lens, the differences would be limited by 1MP and thus not very big, but on a 50MP sensor, the differences should become huge.

Yet.... The opposite is being reported in these tests.

???

No, that is not it.

You´re right about the low resolution sensor, where you´ll see little difference, because of the limiting performance of the sensor. But with the 5DSR you are approaching the maximum performance of the lens and It is the limiting factor. So the performance curve for a lens-sensor combination, with sharpness on the Y-axis and sensor resolution on they X-axis will get an S-shape. The 5DIII sensor resolution will most likely be somewhere close to the steepest part of the curve.

 

[mistaspeedy]

A possible explanation is the ability of the sensor to resolve fine detail.
What if the 5DsR and 5Ds sensors can only resolve detail up to about a score of 1700 and 1600 respectively ?
Think of this as a 'cap' or max value, no matter the lens.
Therefore, on the 300mm lens, they have no problem resolving all of the detail the lens gives them, because it is lower than the values of about 1600-1700.
But when presented with an extremely high quality lens like that Otus, they have hit their limit in resolving power.
The sensors have hit their resolution ceiling and are not resolving significantly more than they do on the 300mm lens.
Now for some calculations!
The Otus is resolving 23.8% more detail on the 5D mark III than the 300mm F2.8.
If we take that 23.8% value and apply them to the 5Ds and 5DsR numbers for the 300mm lens, we should get the following (what the 5Ds and 5DsR 'should be resolving')
5Ds with 85mm Otus: 1702 score [match below with 1701 score]
5DsR with 85mm Otus: 1912 score [match below with 1912 score]
I further confirm these two scores with the following 3 pieces of information listed on the site:
5D mark III with 300mm F2.8 = 1050
5DS with 300mm F2.8 = 1375 = + 30.9% over 5D mark III
5DsR with 300mm F2.8 = 1545 = 47.1% over 5D mark III

Let's now apply these 30.9% and 47.1% differences to the 5D mark III score for the Otus:
1300 + 30.9% = 1701 score
1300 + 47.1% = 1912 score
I'd say those numbers match up nicely!

So I have gone two different routes and gotten the same numbers.
I used the 5D mark III to compare 2 lenses = 23.8% difference
I used the scores on all 3 cameras with the 300mm lens to measure sensor resolving power... it seems that the 300mm lens is sharp enough to show these differences, but not sharp enough to run into a sensor limit.
What about the other Zeiss lens?
5Ds = 27.8% better than 5D mark III
5dsR = 41% better than 5D mark III
What about the Canon 50mm F1.4 ?
5Ds = 19% better than 5D mark III
5DsR = 21% better than 5D mark III

CONCLUSION:
The better the lens, the bigger the differences between sensors! [unless sensor limited]
With the 5DsR, we have:
21% better than 5D mark III on Canon 50mm F1.4
41% better than 5D mark III on Zeiss 21mm F2.8
47.1% over 5D mark III on Canon 300mm F2.8
The 85mm F1.4 Otus did not show these larger differences because we have hit a sensor-resolving limit.
Thank you Andyx01 for bringing this to my attention... you are all free to double-check my math, my logic or to bring your own conclusions!

 

[Andyx01]

Re: S-shaped

The resolving power of a lens is a fixed thing. It is what it is. It wouldn't plot out as a curve. (unless you were referencing from the center to a corner.)

Pretend the sensor had an infinite number of pixels, or something like a billion MP. The lens projection onto the sensor is what it is; it doesn't change. As the resolution of the sensor is reduced from a billion MP to 1MP the amount of detail you can record will be reduced as well. Plot the reduction on a chart and you would end up with a curve, (not an S curve.) The steepest angle of the curve would be limited to 45 degrees, this would be the area where resolution is the limit, the curve would go nearly flat as resolution continued to increase because the lens is only delivering so much.

e.g. horizontal = sensor res and vertical = optical res (see dots below)
.
.
.
.
.
.
.


Compare that curve to another curve from a lesser lens, and you would get a similar shaped curve.

As you work toward 1MP the differences would get smaller (not larger.)

That is my point.

The numerical value being reported as 'resolution' in the chart doesn't add up. Something is definatly skewing the results and no one seems to know what it is.

 

[dcm]

Why don't you post your question in the comments on Roger's blog item and get a response directly from the source of the numbers? I imagine he can shed some light on it for you.

 

[Andyx01]

A possible explanation is the ability of the sensor to resolve fine detail.
What if the 5DsR and 5Ds sensors can only resolve detail up to about a score of 1700 and 1600 respectively ?
Think of this as a 'cap' or max value, no matter the lens.

That's actually a good point! (Limiting factors - I'm making an assumption that the test chart isn't limiting things)

I still find it odd however that if a lens CAN score 1300 on a 5D III (which the 85 does) - that a competing lens that is nearly as sharp (on a high res sensor) scores so much lower. I almost want to say the contrast of the resolution is somehow affecting the resolution score. However -- If it was related to contrast (which is obviously better on the stopped down 85 per Bryans chart) it wouldn't explain why the 300mm score would rise significnatly on the 5Dsr (unless the contrast was less of a factor given the way the scoring system biases total resolution?)

It would be nice to know exactly how these scores come to be.

Like you point out though, perhaps given a better chart, the otus would score substantially higher? (the reason I say limiting chart, and not limiting sensor is because of the large discrepancy between the 5D III sensor and those two lenses.) The 5D III sensor is clearly limiting, yet the score difference is dramatic enough to take note. (esp considering a visual comparison (at least to my eye) shows no resolution difference, but it does show a contrast difference.)

If contrast is indeed the answer - how is the score weighted?

 

[mistaspeedy]

Well, we both have different views on that 300mm lens from Canon... I straight up believe that the Zeiss is simply much better. There is only so much Canon can do with a lens that big... it is simply huge and costs a lot to make having that much glass, having autofocus, having image stabilization, having weather sealing.... the Otus does not have all these things, and the money can be invested into optical quality.

The HUGE Canon 300mm F2.8 costs $6100
The Otus 85mm costs $4500.

 

[Andyx01]

My (unconfirmed) take away from this is:

a) Resolution is measured and assigned a score (up to the resolving power of the test image and sensor.)

e.g. perhaps the 300, and 85 score the same score for for part (a) above.

b) Contrast is measured, and score is reduced accordingly

e.g. the 85 has better contrast and the score isn't reduced as much

Result: The 5D III the scores end up 1050 for the 300mm, and 1300 for the 85mm. (difference due to contrast)



Now increase the sensor resolution by going to the 5Dsr:

a) resolution is again measured but this time perhaps the 300 scores better then the 85 as they are no longer sensor limited.

b) the added resolution has no effect on the vignetting, and again the 300 takes a bigger hit than the 85 does.

The end result still goes to the 85, but by a narrow margin as the 300 scores higher in the resolution test before taking a contrast hit.



This is conjecture of course, but it's the only thing that makes sense. (although even this doesn't make a lot of sense when you bring corner scores into it *sigh*)

 

[Andyx01]

Well, we both have different views on that 300mm lens from Canon... I straight up believe that the Zeiss is simply much better. There is only so much Canon can do with a lens that big... it is simply huge and costs a lot to make having that much glass, having autofocus, having image stabilization, having weather sealing.... the Otus does not have all these things, and the money can be invested into optical quality.

The HUGE Canon 300mm F2.8 costs $6100
The Otus 85mm costs $4500.

The 300mm actually scores better then the Otus 85 though. (Only the center falls 7.4% short, the corner is actually scoring 18.7% better.) The average score goes to the 300mm by 4.8%

Not only does it score better, but it visibly looks better wide open compared to the Zeiss stopped down 2 stops.

Lucky for the Zeiss it's not the same focal length, or zeiss would be in serious trouble. (Edit: Actually they wouldn't be in trouble, I think enough people buy things based on how they feel physologically based on advertising, etc.)

But this brings up another point. (at f/2.8 )

The 85 scores better on a resolution limited body (1015) then the 300 does (920)

-however-

The 300 scores better on a high resolution body (1300) then the 85 does (1240)

Which goes back to the contrast v.s. resolution scenario.

What do you guys think?

 

[mistaspeedy]

Well, my comments were for the 'center' sharpness mentioned in the beginning.

As far as these results you mention... the 300mm is simply sharper in the corners, bringing the average up.
The average for the Otus is being limited by the sensor.. since those 'center' scores should be significantly higher.

In short... they are both excellent lenses, with the Otus having better center sharpness, and the Canon having better corner sharpness.

 

[longtallkarl]

one thing to remember is that image resolution is a function of lens and sensor (someone else can supply the actual equation if they wish - i just know i've read it on other threads.) so even if a lens scores a 1300 on a high resolution sensor that doesn't mean that it will on a lower resolution sensor, even if that sensor is capable of generating a 1300 with a higher quality lens.

just like a higher resolution sensor will get more image resolution out of any quality lens, so too will the image resolution always fall when moving to a lower resolution sensor.

i think mistaspeedy has provided the correct answer.

 

[Andyx01]

one thing to remember is that image resolution is a function of lens and sensor (someone else can supply the actual equation if they wish - i just know i've read it on other threads.) so even if a lens scores a 1300 on a high resolution sensor that doesn't mean that it will on a lower resolution sensor, even if that sensor is capable of generating a 1300 with a higher quality lens.

just like a higher resolution sensor will get more image resolution out of any quality lens, so too will the image resolution always fall when moving to a lower resolution sensor.

i think mistaspeedy has provided the correct answer.

correct answer for what?

 

[longtallkarl]

this:

CONCLUSION:
The better the lens, the bigger the differences between sensors! [unless sensor limited]
With the 5DsR, we have:
21% better than 5D mark III on Canon 50mm F1.4
41% better than 5D mark III on Zeiss 21mm F2.8
47.1% over 5D mark III on Canon 300mm F2.8
The 85mm F1.4 Otus did not show these larger differences because we have hit a sensor-resolving limit.
Thank you Andyx01 for bringing this to my attention... you are all free to double-check my math, my logic or to bring your own conclusions!

 

[bchernicoff]

Another thought. Could false detail be an issue? Here is a quote from the Amateur Photographer review of the 5DS R (http://www.amateurphotographer.co.uk/reviews/dslrs/canon-eos-5ds-r-review/7)

"As we’d expect from a 50MP sensor with no optical low-pass filter, the 5DS R gives remarkable results in our resolution tests. At ISO 50 it resolves very close to its theoretical maximum of 5600l/ph, although with some aliasing and false colour around this point, and plenty of false detail at higher frequencies."

Is it possible Imatest is getting confused? Lensrentals says in the comments that they decided not to publish Otis D810 numbers because: "There was a big enough disparity between the expected result on the Canon and Nikon systems that it through big red flags, so we did not publish that data."

This could be an indication that they are missing something important with their testing methodology.

 

[Andyx01]

Another thought. Could false detail be an issue? Here is a quote from the Amateur Photographer review of the 5DS R (http://www.amateurphotographer.co.uk/reviews/dslrs/canon-eos-5ds-r-review/7)

"As we’d expect from a 50MP sensor with no optical low-pass filter, the 5DS R gives remarkable results in our resolution tests. At ISO 50 it resolves very close to its theoretical maximum of 5600l/ph, although with some aliasing and false colour around this point, and plenty of false detail at higher frequencies."

Is it possible Imatest is getting confused? Lensrentals says in the comments that they decided not to publish Otis D810 numbers because: "There was a big enough disparity between the expected result on the Canon and Nikon systems that it through big red flags, so we did not publish that data."

This could be an indication that they are missing something important with their testing methodology.

Yeah something is definately not right. The fact that they don't post certain results because of a dramatic disparity with expectations along with scores that don't add up, really make me wonder.

Reminds me of Monitor Calibration scores.

I've seen horrible screens score dramatically higher then amazing screens due to the values assigned to where some of the plot points were.

If you buy into the score, you can increase your score by re-mapping look up tables.

Even though you now score higher with the adjusted table - the actual image quality suffered dramatically.

Without knowing how they are scoring these, along with the amount of objectionable data, leads me to question the value of these results.

Thanks for the post - makes me feel better about the wonky data

 

[Andyx01]

this:

CONCLUSION:
The better the lens, the bigger the differences between sensors! [unless sensor limited]
With the 5DsR, we have:
21% better than 5D mark III on Canon 50mm F1.4
41% better than 5D mark III on Zeiss 21mm F2.8
47.1% over 5D mark III on Canon 300mm F2.8
The 85mm F1.4 Otus did not show these larger differences because we have hit a sensor-resolving limit.
Thank you Andyx01 for bringing this to my attention... you are all free to double-check my math, my logic or to bring your own conclusions!

Keep in mind the Canon lens leveraged the increased resolution much better than the Otus on these tests. This would suggest it is the better performing lens as it was impeeded by a larger percent.

Additionally, if the sensor was the limiting factor, the two lenses would have been much closer on the 5D3. The fact that they weren't closer means something other than resolution was boosting the Otus score.

For example, the center scores:
_________ 300mm _________ 85mm_________ 85>300 result.
5d-III_____1050___________1300___________23.8%
+31% +21%
5Ds_______1375___________1570___________14.2%
+12% +6%
5Ds-r______1545___________1660___________7.4%




v.s. The corner scores:
_________ 300mm _________ 85mm_________ 85>300 result.
5d-III_____895___________835___________-7.2%
+20% +16%
5Ds_______1070___________970___________-10.3%
+10% +2%
5Ds-r______1175___________990___________-18.7%



The ever increasing difference as the sensor density increases is how the center table should look as well.

The larger gains on the Canon mean it's leveraging the sensor better. The fact that the baseline score of the Otus is higher suggests another factor is boasting it up (less vignetting due to being stopped down? / contrast?)

 

[bchernicoff]

Whoever suggested you post your concerns in a comment on the lensrentals blog post was right. I'd like to see if they respond.


No registration required to comment: http://www.lensrentals.com/blog/2015/06/canon-5ds-and-5ds-r-initial-resolution-tests