Hi All,
This is Brandon from OLAF Optical Testing. I wish to shed some light on this discussion and hopefully put it to rest.
First and foremost I would like to address the suspicion as to why we tested against the D810 but did not post the results.
The system that produces these results contains many elements which cannot be separated from each other, in order we have the chart, the lens, the operator, the imaging sensor, the camera's internal processing, the raw converter, and the software.
The chart is hopefully not a limiting factor. Honestly I do not know if it is, Roger may know for sure, but I do not think it is if it is printed at at least 300ppi knowing its size. The lens, operator, sensor, raw converter, and software are all limiting factors and are imperfect.
Skipping over the lens briefly, consider the sensor which is itself a system. First we have the stack which contains the color filter array, IR and UV cut filters, and the optical low pass filter. Someone here has wondered why canon goes through the effort of canceling the OLPF instead of removing it altogether - I will explain that first.
An OLPF is traditionally made by using a thin piece of a birefringent material. These materials cannot be coated to reduce reflections and have relatively irregular surfaces which can cause quite a bit of scattering. The most suitable materials are only bifringent in one plane, so you can thick of it as splitting "1" into "2" vertically oriented initially. The IR/UV filter is 'plain' coated glass and can have its reflectance reduced to ideal levels, so it comes next. After it comes the second layer of the OLPF. In a 'functional' OLPF the same trick as the first layer is used, but it is oriented in the other axis, turning vertical "2" into square "4." For a canceled OLPF as in the 5Dsr they essentially install the front filter upside down to cancel the first filter. This is imperfect, but it keeps the optical path length the same which is critical.
If you have a converging beam, as exists between the lens and its capture device, the path length for the "center" ray is shortest and the path grows longer as you look to the more peripheral sections of the beam. If you look at this lens diagram - http://i.imgur.com/j4wYtY7.png - and specifically the blue rays, the upper and lower ones have a longer path than the middle one. The OPL of the sensor stack is not constant across manufactures, this is a major source of error when comparing the same lens model across camera brands - we do not know which thickness the lens is designed for.
If you alter the optical path length (which is essentially a form of "apparent length" to light) of the sensor stack, those upper and lower rays will not form the same common focus they used to form with the middle ray, and you get spherical aberration. Thus, keeping the OPL constant is important. The CFA has a simply impact on MTF from a practical standpoint, from a physical standpoint it doesn't have one.
Ignoring the color filter array, which is cemented to the sensor itself and cannot easily be removed, the sensor itself then has a "sum" MTF. The sensor MTF, like a lens' MTF, may be represented as a fourier transform and we can plot contrast vs frequency. In all imperfect systems the MTF will be curved - not linear. If you look at fig 2.3 here - http://spie.org/x34358.xml - note that the sensor MTF hits 0 at a particular frequency and then bounces off zero for a bit. This false detail is what produces aliasing, etc. An ideal OLPF would "brickwall" filter at the zero point, but this is an impossible goal to realize in terms of manufacturing if not design.
The system MTF is simply the multiple of its parts, so SensorMTF*LensMTF=SystemMTF. This is essentially what is being sampled by programs like Imatest (though this is a gross oversimplification).
Worth noting here that films also have their own MTF curves, as sensors do.
From there you get to the raw converter, which does some form of processing to turn a patterned grid of red-corresponding, green-corresponding, and blue-corresponding dots into full RGB matrix dots. This is an uncontrollable factor without writing your own software - the truth is we don't know how dcraw, embedded in Imatest, Canon's DPP software, Adobe, etc, do this except that they use the "best" method they can come up with. This is a major source of error, even if the converter is standardized to a single software.
Moving onto the software - the first stage of using Imatest is to run a software calibration and an alignment calibration. The 300 II is nice here because the camera mounts to it instead, so tilt, etc, usually do not need to be adjusted between cameras. This returns an alignment error quantity - our standard is to have <0.1deg of tilt. We know tilt will damage the result (another source of test error) but we do not know how much, and imatest cannot tell us because they are also unaware of how much difference it makes, is it linear, etc.
Once we get past this result, the software calculates a contrast ratio based on the spread of lines oriented along various angles w.r.t the sensor normal. There are several algorithms internal to the software which adjust the raw number to calibrate various errors in the calculation method and known phenomena, but again they are not something we can control. We do not endorse Imatest or claim that its calculation method is bulletproof, the best method available, etc. These calibrations if not done properly are another source of error. Lenses approaching the diffraction limit should require different corrections than those very far from it - this is a very large source of imprecision.
----
Returning to the lens - the lens' MTF is calculated via a fourier transform of a known source. In the case of design software, the spot is simulated and gets FFT'd. In this article I wrote for photozone recently you can see a manufacture curve in the internal industry standard format - http://www.photozone.de/aberrationsExample
Essentially MTF vs freq gives more information than sampling MTF at specific frequencies and plotting against image height, so we use a different format.
Note that the curve - especially the blue curve - bounces off of zero. This is false resolution due to phase error in the lens. All non-diffraction-limited lenses do this. The better the lens, the higher the spatial frequency it happens at. Usually it is higher than the spatial frequency of the sensor being used. More importantly, the curve is not linear and without knowing it explicitly, how a lens "scales" to different sensors is not something one can even begin to hypothesize accurately. If the 5D3 and 5Ds were both sampling at a spatial frequency in the large, flat, near-zero region of that graph the lenses would appear to improve linearly as sensor res rises, but even the rather soft EF 50mm f/1.4 flattens out like that at a spatial frequency higher than the 5D3 can sample.
With the D810, we basically saw an extremely unnatural scaling between D810:5Dsr with the otus that was not expected. Due to dcraw not being updated and adobe camera raw not being updated, we had to use DPP to process the 5Dsr files and could not guarantee processing integrity. We also do not know which stack thickness the otus is optimized for and therefore which camera has more "lens error" - so the results are invalid anyway.
Between the 5D3 and 5Dsr we are also using different converters, dcraw for the 5D3 and DPP for the 5dsr. In terms of operator error we are using focus bracketing, MLU, etc, but thousands of lenses move through the room this test is done in every day and there are lots of vibrations that can muck with things. The 300 measures better with IS on consistently than with it off. Even with focus bracketing we cannot achieve precisely the same focus between bodies, only that we are at the best of several sample focus points. I.e nuanced differences should not be considered.
This is why we did not post them, nor do we really ever intend to. Speaking frankly and in my own opinion, Imatest and all other camera-based lens analysis tools are crude and poor. We often measure lenses on the MTF bench or OLAF and they look terrible - but they look as good as any "good copy" on camera because the camera is not sensitive enough. Imatest is only useful to us now for comparing cameras, but it is deeply flawed for that purpose. (end opinion).
In terms of Otus vs 300mm comparison, here is the plotted raw data that we used to determine the otus on the 5Dsr was a "good copy" - https://dl.dropboxusercontent.com/u/39289306/TestOtus.pdf - this is at f/1.4, as are all our MTF measurements for the foreseeable future. Here is a different otus that tested worse, so we did not use this copy - https://dl.dropboxusercontent.com/u/39289306/ExampleOtus_2.pdf.
Regarding Amateur Photographer.co.uk - we know little about their testing. We know that:
* if they are backing up between cameras, they are not using a frame-filling chart or their chart isn't high enough res, so already there is a source of error
* refocusing the lens, esp. with asymmetrical (not aspheric, asymmetrical) internal focusing lenses will change performance
* are they using sharpening?
* The non-L 100mm USM is a poor choice of lens for testing cameras with - at the nyquist of the 5Dsr the Zeiss 100mm makro for instance delivers over 2x the resolution on an MTF bench, and has lower sample variation to boot.
* do they have a "good copy" for this - how do they know?
Our bench measurement are made without any kind of flat in the path, so they do not reflect 1:1 performance on camera, but they do allow us to compare all lenses with "infinite" sampling resolution. This particular 4-rotation chart is mostly internal. They need to be too big to show on the blog, but they allow us to very quickly determine if a lens is tilted, etc, so we can verify that every single copy we average is a "good copy". Soon we are launching a new measurement set that we have recorded 220 different lens copies for as of today and will expand by over 100 copies per week at least through august. We will present graphs in this format - average MTF - https://dl.dropboxusercontent.com/u/39289306/EF%2014mm%20f2.8_avg_.png - and the variance plot w/ score - https://dl.dropboxusercontent.com/u/39289306/EF%2014mm%20f2.8_var_.png. We do not yet have enough measurements of different otii to comment on that lens, but this is a $2,000 canon L lens to give you an idea of how big the range between copies can be. Bad copies are already removed from this. I will not explain the charts much - that comes soon during the launch post. Lots of computation goes into the variance plot and it is too much to explain now.
"Will we retest at aperture xxx, with lens yyy, etc etc" - Roger Aaron and I will not, it is not happening. Imatest is very frustrating to work with because there is lots of math done behind the scenes that is not transparent, too many sources of error are not well understood by anyone using the software - not even its makers, and setting up the tests takes forever.
Right now there are over 9,000 items waiting to be inspected during peak season and we have several inspection techs out. We cannot afford the time to test more, and we cannot guarantee result accuracy because of the flaws in camera-based testing.
As frequently as we can we are testing lenses on the MTF bench. It is a "pure" testing methodology and we have control over everything necessary to guarantee accuracy down to +/-0.005. In the range of 0-1.0, this is astronomically smaller than what is possible via something like imatest.
Please let me know if you have any other questions about the test, future numbers, our methods, etc, and I will answer as best I can.
Warmest regards,
Brandon
OLAF Optical Testing
