This is pretty straightforward. The Canon LENS is better than the Nikon LENS. They got the same LENS score. Yep, pretty much sums it up.
Upvote
0
Dxo tests canon/nikon/sony 500mm's
- Thread starter Apop
- Start date
- Status
Pi said:
Umm, no...sorry. The final image is a convoluted result...one could not extract a "pure" lens resolution...you could only approximate it. (For the very same reason one cannot perfectly extract noise from a noisy image...it is part of a convolution produced by a complex real-world system. Too much uncertainty and a loss of information prevents perfect noise removal.) A mathematically generated MTF that takes into account the real mathematical point spread function of the entire lens is really the only way to get any realistic idea of how a lens will actually perform. The moment that convolution is further convolved by a sensor, you lose the ability to "perfectly" (or purely) revert to the prior result...there is too much uncertainty and loss of information.
Pi said:
One wouldn't, necessarily. But your missing the point. The point is to call out DXO's BS approach to performing lens tests. The point is to clearly note that those tests are "camera system" tests...they are neither lens tests nor sensor tests. I wouldn't go so far as to say that is 100% useless, but it is certainly biased the way DXO does it, and there is a suspiciously long-term bias towards a particular manufacturer by DXO. (Not just away from Canon, either...even the Sony lens, which actually has better transmission, should have scored better...but it was limited by a sensor!)
Upvote
0
jrista said:
You are wrong on that. I am not saying that you can remove the AA filters/sensor blur from the image. I am saying that you can find (estimate, if you wish) the strength of the sensor blur. If you are interested in the math, go to my profile, click on the link, etc. Deconvolution is a very different process, very unstable but you do not need to deconvolute to estimate the effect of the sensor blur. You can get instability only if you use sensors with such a low resolution, that the lenses you want to compare look the same (and they are not).
The problem with all that is that even if you are going to get the pure lens resolution somehow, you still need to consider the blurring effect of a future sensor, and compute the combined resolution again. So my question stands: are you sure you know how to do that?
Of course those are lens+camera tests, and DXO never said otherwise. They sometimes write funny articles, and I simply stop reading them. But the numbers are meaningful; or should I say were meaningful before they decided that we are too stupid to understand what MTF meant and decided to use an undocumented metric.
Upvote
0
Pi said:
1. That assumes I'll get a 5D4. Maybe I'll re-evaluate my entire system if data suggest that another brand is better. Good data allows me to make the right decision for my needs.
2. Wait! SOMEONE is WRONG on the Internet!!! (http://xkcd.com/386/)
It's irrelevant to me at this point since I can't justify the cost of any of these lenses for my amateur needs. I simply like to see honest, accurate information. Misleading data (or misleading presentation of data) makes me feel cheated.
Cheers.
Upvote
0
Pi said:
It doesn't matter what kind of sensor you have, low resolution, high resolution, or tomorrows resolution. A convolved result is a convolved result, and in this case stability (or the lack thereof) doesn't really apply like it might when trying to denoise or deblur. You are talking about reverse engineering the actual lens PSF from an image produced by a grid of spatially incongruent red, green, and blue pixels (likely covered by additional lenses (microlenses)), then further interpolated by software to produce the kind of RGB color pixels we see on a screen and analyze with tools like Imatest (or DXO's software). The moment you bring the sensor into play, there are significant enough losses of data, and you can only, at best, guess at what those losses are (unless you have some detailed inside knowledge about whatever sensor it is your testing with). Your article is an interesting start, but you are assuming a Gaussian PSF. An actual PSF is most definitely not Gaussian, nor is it constant across the area of the lens (i.e. it changes as you leave the center and approach the corners...do a search for "spot diagram" to see actual lens PSF's produced mathematically from detailed and accurate lens specifications...even for the best of lenses, outside of the most centeral on-axis results, a PSF can be wildly complicated). Not to mention the fact that you have to guess the kernel in the first place, so whatever your result, it is immediately affected by what you think the lens is capable of in the first place.
Personally, I wouldn't trust any site that provided "lens resolution" results reverse engineered from an image produced by any sensor. I would actually rather take the "camera system" tests than have someone telling me what their best guess is for lens performance.
Pi said:
Hmm, DXO's own description on the lens tests page begs to differ:
Nowhere in there do they state that the camera sensor is a factor in your ability to select and compare lenses. They only state that the lens characteristics, brand, type, focal range, aperture, and price are the applicable factors.
Upvote
0
jrista said:
Let me repeat. The question here is NOT to reconstruct the image before the convolution. It is to reconstruct the convolution kernel knowing what the image was and knowing what the convoluted image is. This is a very different problem, and a well posed one. In the case under discussion, we have two kernels but we have more than two bodies, so you get a system, etc.
Why do not read their description first? They do not demosaic, and they test on each channel. Unfortunately, they decided to hide the data since recently. They do the slanted edge test, which averages over many pixels and makes it possible to estimate well what the effect of the pixels is just based on their number (but not the AA filter strength). It is the "purest" test I have seen but again, the data is hidden now.
I did say that it is not a Gaussian but for many purposes it is close enough. This is not my formula,
it first appeared in a paper on optics and has been used many times since then. If somebody can point out a reference, I will put it there right away.
I also explained why its variance across the frame is not a problem. You just apply the formula in different regions with different values.
I also mentioned there that my point is not that formula but the general principle: multiple blur factors act as a convolution with their convolution. This is a much more universal principle.
There is no much go guess. The AA filter can very well be approximated with a Gaussian, and the effect of the pixels can just be computed. The only weakness is that you do not really know that the AA filter has strength proportional to the pixel density. But then you have data for many bodies.
But even if you had the absolute lens data, what are you going to do with it? You never answered that question. Let me help you - to see how it performs on a future 5D4 or whatever, you need to make some assumptions on the AA filter, then you need to apply the formula that you do not like. There is no way around that factor.
You probably never go to MRI or CT scan test because instead of just "seeing" what is inside, they "reverse engineer", i.e., they compute it.
This is nitpicking. Try to get the resolution numbers. You cannot get it without choosing a body, and the results are always displayed with the body well visible. Their articles are poorly written but it is not a "rocket science" to realize what you are looking at.
Upvote
0
ankorwatt said:
I'll take this page showing real ISO crops over some MTF lab test anyday I'm going to buy a lens to take pictures with:
http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=458&Camera=453&Sample=0&FLI=0&API=0&LensComp=648&CameraComp=0&FLIComp=0&APIComp=0
I'll worry about the "real MTF" test next time I buy a lens to take it to the lab.
Upvote
0
I personally think this is the weirdest lens test article I have ever read. It is not a lens test, but a system test.
What it comes to Hasselblad measuring the MTF of the superteles, not all information is included there. Typically reduced secondary spectrumdoublets achromats have a better MTF than equal element number apochromatic designs, but the color error is then visible with areas of high contrast difference. I'm also interested in the Hasselblad's methodology of doing the test, could you send me the link, please?
It is not always clear to me whether Nikon is reviewed with software corrections enabled on the color aberrations in these tests or not. For that reason, I'd be a bit suspicious on using current Nikon objectives with film.
Even in the Digital Pictures crops, it is not clear to me, how the crops were obtained from RAWs. But for me it seems that the Canon lens performs actually better than Nikon's at this distance for all tested field points. Whether this difference is significant in real life photography is another story.
By looking at the manufacturers MTFs for 200/2.0s, it seems Canon has optimized the 10 lp/mm performance while slightly sacrificing 30 lp/mm performance compared to Nikon, and averagely the impression seems to be that the Canons 200 mm performs better (to my eye), and considerably better close to the edge. This is nothing new, it is known that images taken with a lens that has a better macro contrast tend to look better than the ones taken with a lens with less contrast but higher micro-contrast. Only in the center crop we can see a bit of the micro-contrast at play, but it isn't clear to me what causes aliasing (or Moire?) on the Canon crop, is it the lens or the sensor.
The MTF tests that we do actually does not include the camera body, but a microscope objective and a known image sensor. It is arguable whether this gives full information since there is further processing due to the software, but it allows checking which lenses are better corrected to begin with.
What it comes to Hasselblad measuring the MTF of the superteles, not all information is included there. Typically reduced secondary spectrum
It is not always clear to me whether Nikon is reviewed with software corrections enabled on the color aberrations in these tests or not. For that reason, I'd be a bit suspicious on using current Nikon objectives with film.
Even in the Digital Pictures crops, it is not clear to me, how the crops were obtained from RAWs. But for me it seems that the Canon lens performs actually better than Nikon's at this distance for all tested field points. Whether this difference is significant in real life photography is another story.
By looking at the manufacturers MTFs for 200/2.0s, it seems Canon has optimized the 10 lp/mm performance while slightly sacrificing 30 lp/mm performance compared to Nikon, and averagely the impression seems to be that the Canons 200 mm performs better (to my eye), and considerably better close to the edge. This is nothing new, it is known that images taken with a lens that has a better macro contrast tend to look better than the ones taken with a lens with less contrast but higher micro-contrast. Only in the center crop we can see a bit of the micro-contrast at play, but it isn't clear to me what causes aliasing (or Moire?) on the Canon crop, is it the lens or the sensor.
The MTF tests that we do actually does not include the camera body, but a microscope objective and a known image sensor. It is arguable whether this gives full information since there is further processing due to the software, but it allows checking which lenses are better corrected to begin with.
Upvote
0
ankorwatt said:
Never said it was, just tried to make the point that MTF does not tell the entire story and I prefer equipment that produce best images to my eyes and couldn't care less about numbers from a lab. And the Nikon 200f2 looks softer in the corner compared to Canon on those actual images.
Upvote
0
msm said:
Not that I disagree with you about the visual evidence vs. the lab results ... but if your goal is to get the whole PSF (the image of an ideal point), then the MTF is giving essentially its Fourier transform, and from there, you get the PSF in a direct and a stable way. This does not work is such a straightforward way when the PSF is too concentrated near a single pixel. Still, the whole MTF is in some sense the "complete data" that DXO is hiding from us.
Upvote
0
msm said:
I think you are talking about something different: the sampling theorem (still connected to Fourier analysis). The sampling theorem still applies to "sampling" by integration over the whole pixel (you made a very good observation) but it needs to be modified. Imagine the whole image first convoluted with a pixel (a function equal to 1 in a square of the size of a pixel, zero otherwise). This convolution is band limited (same limit), if the original image is. Then you point sample that.
Upvote
0
As far as I know, digital image sensors are a bit more complicated case than the classical sampling theorem would predict. First of all, it is important to understand the full meaning that the captured image is a two three dimensional signal (x, y and intensity) and how the eye sees it.
Using classical sampling theorem, a maximum resolvable frequency could be found by taking the inverse of (2*pixel pitch), which would lead to Nyqvist cut-off frequency. However, this is not the case, as in the measurements the image sensor tends to see further, as explained in [1] and published in [2]
As a short version, if one is able to align the pixel array exactly in the direction of bar patterns, the classical Nyqvist frequency holds. However, it is very difficult to do this, and thus what is actually seen is a result of sub-pixel sampling, which is then averaged by the eye and interpreted as a distinguishable bar. If one would only take a single line of the image, I'm not sure if the result in that case would be classified as distinguishable.
Add on top of that the fact whether we want to represent the actual shape of the subject at the maximum resolvable frequency despite the fact if it lands between the pixels, it can be seen that there can be a need for three to five times oversampling. I don't unfortunately have a good link to show this, I'll try to look for it and post it whether I can find it. However, this tends to be a way of selling more pixels too.
EDIT: Ah, found it, the PDF was by Andor [3]. What I want to say with all this, is that it is actually not that well defined what is meant by "resolving something" with the image sensors.
Using classical sampling theorem, a maximum resolvable frequency could be found by taking the inverse of (2*pixel pitch), which would lead to Nyqvist cut-off frequency. However, this is not the case, as in the measurements the image sensor tends to see further, as explained in [1] and published in [2]
As a short version, if one is able to align the pixel array exactly in the direction of bar patterns, the classical Nyqvist frequency holds. However, it is very difficult to do this, and thus what is actually seen is a result of sub-pixel sampling, which is then averaged by the eye and interpreted as a distinguishable bar. If one would only take a single line of the image, I'm not sure if the result in that case would be classified as distinguishable.
Add on top of that the fact whether we want to represent the actual shape of the subject at the maximum resolvable frequency despite the fact if it lands between the pixels, it can be seen that there can be a need for three to five times oversampling. I don't unfortunately have a good link to show this, I'll try to look for it and post it whether I can find it. However, this tends to be a way of selling more pixels too.
EDIT: Ah, found it, the PDF was by Andor [3]. What I want to say with all this, is that it is actually not that well defined what is meant by "resolving something" with the image sensors.
Upvote
0
Mika said:
Those links have nothing to do with the sampling theorem. The latter does not care whether you image bars, etc., it tells you how to sample an a priori band limited signal (the bars are NOT that), and how to reconstruct it. The modification needed that I mentioned is simple and must have been done by somebody already. In short, if your image is band limited already (this is what the AA filter does, together with the lens), and you have a good estimate what that limit is, you know how many pixels you need.
Do not confuse a convenient resolution test (bars) with the sampling theorem.
Upvote
0
Pi said:
There is a misunderstanding somewhere here, for me it sounds like we are talking about different things or use different terms. I'm well aware of the different nature of the problem described in [3]. However, what I meant to say with that is related to your earlier PSF considerations, when characterizing the PSF, the energy in the typical photographic objective spot is typically within the region of 1-3 camera body pixels, with a central core of the energy (something like 80 %) in a single pixel.
So in that case, you would be quite subject to errors in estimating the PSF due to the effect shown in [3]. And you really don't know the PSF beforehand. Only at the proximity of image edges (or using fast lenses) the PSF may become large enough to be sampled well by the camera sensor. If you are using a different bench for estimating the PSF with magnification, you'll then lose the effect of the AA filter as well.
Also, the photographic objective MTF isn't typically evaluated from a PSF (haven't seen this being used in many places), but from an edge or line spread function which then allows sub-pixel sampling and is more robust against positioning with respect to sampling grid. Astronomical telescopes may be a different thing, I don't have experience in designing them.
The point of [1] was to show that for example, depending on the angle the camera is mounted with respect to the bar chart target, your micro-contrast figures may change slightly.
None of this actually matters to the actual photography, though. I don't know whether we should continue with private messages, I suppose this is going to get technical and lots of people aren't probably interested in seeing this.
Upvote
0
Mika said:
I love this kind of stuff, and have been reading your discussion so far. Instead of private messages, maybe just start a new thread, and link to the conversation here? So far, my understanding is more in line with yours, Mika...but I'd like to see what Pi has to say on the subject, as perhaps there is something new to learn.
Upvote
0
Mika said:
Exactly. The slanted edge test averages over a relatively long edge, and I do not know of a single test which would try to look at a single point, or would try to align the bars exactly, etc. But that allows you to restore the actual PSF (convoluted with something but let us keep it simple) by a simple calculation.
It matters enough for so many labs and companies to measure the MTF, for academics to write papers and books, etc.
Upvote
0
Sorry about the delay in replying, the weather has been (almost too) good in this week.
What it comes to slanted edge testing, this is where I disagree (partially). If we consider a slanted edge test with a body+lens setup, there are several issues in that what I'd think as a deal breaker for recovering the real point spread function as I know it.
First, the pixel pitch typically does not actually support sufficient sampling. Second, the slanted edge is considerably larger and thus the average of the line spread functions is taken over a comparatively large image block where PSF has probably changed by some amount - this is typical for wide angle constructs where there are several aspherical surfaces. And if the length of the slanted edge isn't long enough, there will be uncertainty in the slant angle and the sub-pixel sampling is then affected. Third, given the slant angle is small, this test methodology cannot differentiate between imaging quality of tangential and sagittal axes and can miss changes in the averaging direction completely.
For an extreme example, it would report the MTF of a cylinder lens system equal to a spherical lens system if it was aligned along the imaging axis. This mistake of course, is hard to imagine happening in real life, but extending the thought for a bit, it is easier to understand that decentered elements along one axis could be missed with this. For this reason, lens would need to be turned 90 degrees to determine both directions.
The bar chart quality assurance benches that I have seen are used as OK/NOK step in quality control. The actual MTF measurement benches magnify the known spot with a high quality microscope objective, and thus this measurement of the MTF is much more local, and for that reason I accept it as a representative PSF. The only people who I do know to have sampled the PSF directly are astronomers.
What I'm saying here is not that the slanted edge method in lens+body setup isn't useful in determining MTF (with certain error bounds), it is. It is also very useful in relative comparisons if all systems are measured in the same bench. But what it does not do is provide scientifically accurate MTF values, and additionally, the online reviews are usually about resolving power of a body+lens combination, but the macro-contrast level is not that often reported.
So I suppose it all boils down on what is accepted as a PSF.
What it comes to slanted edge testing, this is where I disagree (partially). If we consider a slanted edge test with a body+lens setup, there are several issues in that what I'd think as a deal breaker for recovering the real point spread function as I know it.
First, the pixel pitch typically does not actually support sufficient sampling. Second, the slanted edge is considerably larger and thus the average of the line spread functions is taken over a comparatively large image block where PSF has probably changed by some amount - this is typical for wide angle constructs where there are several aspherical surfaces. And if the length of the slanted edge isn't long enough, there will be uncertainty in the slant angle and the sub-pixel sampling is then affected. Third, given the slant angle is small, this test methodology cannot differentiate between imaging quality of tangential and sagittal axes and can miss changes in the averaging direction completely.
For an extreme example, it would report the MTF of a cylinder lens system equal to a spherical lens system if it was aligned along the imaging axis. This mistake of course, is hard to imagine happening in real life, but extending the thought for a bit, it is easier to understand that decentered elements along one axis could be missed with this. For this reason, lens would need to be turned 90 degrees to determine both directions.
The bar chart quality assurance benches that I have seen are used as OK/NOK step in quality control. The actual MTF measurement benches magnify the known spot with a high quality microscope objective, and thus this measurement of the MTF is much more local, and for that reason I accept it as a representative PSF. The only people who I do know to have sampled the PSF directly are astronomers.
What I'm saying here is not that the slanted edge method in lens+body setup isn't useful in determining MTF (with certain error bounds), it is. It is also very useful in relative comparisons if all systems are measured in the same bench. But what it does not do is provide scientifically accurate MTF values, and additionally, the online reviews are usually about resolving power of a body+lens combination, but the macro-contrast level is not that often reported.
So I suppose it all boils down on what is accepted as a PSF.
Upvote
0
Mika said:
It does but not of the PSF directly - of the PSF convoluted with "something", derived from the pixel size and the AA filter. That "something" is a known quantity, depending on the angle as well. From there, you can get the PSF. Again, the computation is NOT the same as deconvolution but still, it is not going to be too accurate if the PSF is too concentrated compared to one pixel but the instability is far from the (exponential) one for deconvolution. The lens PSF convoluted with the effect of the AA filter however is not "too concentrated" (the reason AA exists in the first place), and can be well reconstructed. Factoring out the effect of the AA filter itself if trickier but if you keep the same sensor, you want to keep that effect in place.
Look at the actual targets. They consists of squares which are not "too large" but "large enough". In any case, you are measuring some average of the PSF along the edge, hoping that it does not change wildly.
They use black slanted squares in the target. In the good old times, DXO would report horizontal and vertical resolution (and they usually were different enough).
Do not underestimate the power of computed PSF vs. the measured one. In most modern medical imaging techniques, for example, the image is computed from the data with serious math methods, vs. just displaying pictures as with the traditional X-ray tomography, for example. BTW, my work is related to that.
Upvote
0
- Status
