April 19, 2014, 01:18:00 AM

Author Topic: Dxo tests canon/nikon/sony 500mm's  (Read 12284 times)

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #60 on: July 16, 2013, 05:44:26 PM »
Since you can mount the Nikon lens on a Canon body with an adapter, why not compare apples to apples?

You mean, an apple on an orange to an orange on an orange?

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #60 on: July 16, 2013, 05:44:26 PM »

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #61 on: July 16, 2013, 05:51:59 PM »
Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.


Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

Quote
The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

Why in the world would you want to know how a Canon compares to a Nikon without a body? For bragging rights? They tell you what is achievable with the current bodies on which the lens works, the way it is deigned to work. A better lens on one body will be better on future bodies as well.

bdunbar79

  • 1D X
  • *******
  • Posts: 2427
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #62 on: July 16, 2013, 06:11:10 PM »
This is pretty straightforward.  The Canon LENS is better than the Nikon LENS.  They got the same LENS score.  Yep, pretty much sums it up.

jrista

  • 1D X
  • *******
  • Posts: 3239
  • POTATO
    • View Profile
    • Nature Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #63 on: July 16, 2013, 06:23:16 PM »
Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.


Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

Umm, no...sorry. The final image is a convoluted result...one could not extract a "pure" lens resolution...you could only approximate it. (For the very same reason one cannot perfectly extract noise from a noisy image...it is part of a convolution produced by a complex real-world system. Too much uncertainty and a loss of information prevents perfect noise removal.) A mathematically generated MTF that takes into account the real mathematical point spread function of the entire lens is really the only way to get any realistic idea of how a lens will actually perform. The moment that convolution is further convolved by a sensor, you lose the ability to "perfectly" (or purely) revert to the prior result...there is too much uncertainty and loss of information.

Quote
The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

Why in the world would you want to know how a Canon compares to a Nikon without a body? For bragging rights? They tell you what is achievable with the current bodies on which the lens works, the way it is deigned to work. A better lens on one body will be better on future bodies as well.

One wouldn't, necessarily. But your missing the point. The point is to call out DXO's BS approach to performing lens tests. The point is to clearly note that those tests are "camera system" tests...they are neither lens tests nor sensor tests. I wouldn't go so far as to say that is 100% useless, but it is certainly biased the way DXO does it, and there is a suspiciously long-term bias towards a particular manufacturer by DXO. (Not just away from Canon, either...even the Sony lens, which actually has better transmission, should have scored better...but it was limited by a sensor!)
My Photography
Current Gear: Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: Canon 5D III/7D II | Canon EF 300mm f/2.8 L II

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #64 on: July 16, 2013, 06:40:52 PM »
Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.


Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

Umm, no...sorry. The final image is a convoluted result...one could not extract a "pure" lens resolution...you could only approximate it. (For the very same reason one cannot perfectly extract noise from a noisy image...it is part of a convolution produced by a complex real-world system. Too much uncertainty and a loss of information prevents perfect noise removal.)

You are wrong on that. I am not saying that you can remove the AA filters/sensor blur from the image. I am saying that you can find (estimate, if you wish) the strength of the sensor blur. If you are interested in the math, go to my profile, click on the link, etc. Deconvolution is a very different process, very unstable but you do not need to deconvolute to estimate the effect of the sensor blur. You can get instability only if you use sensors with such a low resolution, that the lenses you want to compare look the same (and they are not).

The problem with all that is that even if you are going to get the pure lens resolution somehow, you still need to consider the blurring effect of a future sensor, and compute the combined resolution again. So my question stands: are you sure you know how to do that?

Quote
One wouldn't, necessarily. But your missing the point. The point is to call out DXO's BS approach to performing lens tests. The point is to clearly note that those tests are "camera system" tests...they are neither lens tests nor sensor tests. I wouldn't go so far as to say that is 100% useless, but it is certainly biased the way DXO does it, and there is a suspiciously long-term bias towards a particular manufacturer by DXO. (Not just away from Canon, either...even the Sony lens, which actually has better transmission, should have scored better...but it was limited by a sensor!)

Of course those are lens+camera tests, and DXO never said otherwise. They sometimes write funny articles, and I simply stop reading them. But the numbers are meaningful; or should I say were meaningful before they decided that we are too stupid to understand what MTF meant and decided to use an undocumented metric.
« Last Edit: July 16, 2013, 06:44:09 PM by Pi »

neuroanatomist

  • CR GEEK
  • *******
  • Posts: 12779
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #65 on: July 16, 2013, 06:48:33 PM »
But the numbers are meaningful; or should I say were meaningful before they decided that we are too stupid to understand what MTF meant and decided to use an undocumented metric.

Now, now...be fair.  P-Mpix is just as well documented as many of the other values they publish.   ::)
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Orangutan

  • 7D
  • *****
  • Posts: 419
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #66 on: July 16, 2013, 07:29:12 PM »
But a better lens on, say, the 5D3, will preform better on the 5D4, you know that for sure. Again, where is the problem exactly?


1. That assumes I'll get a 5D4.  Maybe I'll re-evaluate my entire system if data suggest that another brand is better.  Good data allows me to make the right decision for my needs.

2. Wait!  SOMEONE is WRONG on the Internet!!!   (http://xkcd.com/386/)

It's irrelevant to me at this point since I can't justify the cost of any of these lenses for my amateur needs.  I simply like to see honest, accurate information.  Misleading data (or misleading presentation of data) makes me feel cheated.

Cheers.

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #66 on: July 16, 2013, 07:29:12 PM »

jrista

  • 1D X
  • *******
  • Posts: 3239
  • POTATO
    • View Profile
    • Nature Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #67 on: July 16, 2013, 07:31:47 PM »
Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.


Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

Umm, no...sorry. The final image is a convoluted result...one could not extract a "pure" lens resolution...you could only approximate it. (For the very same reason one cannot perfectly extract noise from a noisy image...it is part of a convolution produced by a complex real-world system. Too much uncertainty and a loss of information prevents perfect noise removal.)

You are wrong on that. I am not saying that you can remove the AA filters/sensor blur from the image. I am saying that you can find (estimate, if you wish) the strength of the sensor blur. If you are interested in the math, go to my profile, click on the link, etc. Deconvolution is a very different process, very unstable but you do not need to deconvolute to estimate the effect of the sensor blur. You can get instability only if you use sensors with such a low resolution, that the lenses you want to compare look the same (and they are not).

The problem with all that is that even if you are going to get the pure lens resolution somehow, you still need to consider the blurring effect of a future sensor, and compute the combined resolution again. So my question stands: are you sure you know how to do that?

It doesn't matter what kind of sensor you have, low resolution, high resolution, or tomorrows resolution. A convolved result is a convolved result, and in this case stability (or the lack thereof) doesn't really apply like it might when trying to denoise or deblur. You are talking about reverse engineering the actual lens PSF from an image produced by a grid of spatially incongruent red, green, and blue pixels (likely covered by additional lenses (microlenses)), then further interpolated by software to produce the kind of RGB color pixels we see on a screen and analyze with tools like Imatest (or DXO's software). The moment you bring the sensor into play, there are significant enough losses of data, and you can only, at best, guess at what those losses are (unless you have some detailed inside knowledge about whatever sensor it is your testing with). Your article is an interesting start, but you are assuming a Gaussian PSF. An actual PSF is most definitely not Gaussian, nor is it constant across the area of the lens (i.e. it changes as you leave the center and approach the corners...do a search for "spot diagram" to see actual lens PSF's produced mathematically from detailed and accurate lens specifications...even for the best of lenses, outside of the most centeral on-axis results, a PSF can be wildly complicated). Not to mention the fact that you have to guess the kernel in the first place, so whatever your result, it is immediately affected by what you think the lens is capable of in the first place.

Personally, I wouldn't trust any site that provided "lens resolution" results reverse engineered from an image produced by any sensor. I would actually rather take the "camera system" tests than have someone telling me what their best guess is for lens performance.

Quote
One wouldn't, necessarily. But your missing the point. The point is to call out DXO's BS approach to performing lens tests. The point is to clearly note that those tests are "camera system" tests...they are neither lens tests nor sensor tests. I wouldn't go so far as to say that is 100% useless, but it is certainly biased the way DXO does it, and there is a suspiciously long-term bias towards a particular manufacturer by DXO. (Not just away from Canon, either...even the Sony lens, which actually has better transmission, should have scored better...but it was limited by a sensor!)

Of course those are lens+camera tests, and DXO never said otherwise.

Hmm, DXO's own description on the lens tests page begs to differ:

Quote
DxOMark's comprehensive camera lens test result database allows you to browse and select lenses for comparison based on its characteristics, brand, type, focal range, aperture and price.

Nowhere in there do they state that the camera sensor is a factor in your ability to select and compare lenses. They only state that the lens characteristics, brand, type, focal range, aperture, and price are the applicable factors.
« Last Edit: July 16, 2013, 07:33:33 PM by jrista »
My Photography
Current Gear: Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: Canon 5D III/7D II | Canon EF 300mm f/2.8 L II

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #68 on: July 17, 2013, 01:57:42 AM »

It doesn't matter what kind of sensor you have, low resolution, high resolution, or tomorrows resolution. A convolved result is a convolved result, and in this case stability (or the lack thereof) doesn't really apply like it might when trying to denoise or deblur.

Let me repeat. The question here is NOT to reconstruct the image before the convolution. It is to reconstruct the convolution kernel knowing what the image was and knowing what the convoluted image is. This is a very different problem, and a well posed one. In the case under discussion, we have two kernels but we have more than two bodies, so you get a system, etc.

Quote
You are talking about reverse engineering the actual lens PSF from an image produced by a grid of spatially incongruent red, green, and blue pixels (likely covered by additional lenses (microlenses)), then further interpolated by software to produce the kind of RGB color pixels we see on a screen and analyze with tools like Imatest (or DXO's software).


Why do not read their description first? They do not demosaic, and they test on each channel. Unfortunately, they decided to hide the data since recently. They do the slanted edge test, which averages over many pixels and makes it possible to estimate well what the effect of the pixels is just based on their number (but not the AA filter strength). It is the "purest" test I have seen but again, the data is hidden now.
 

Quote
Your article is an interesting start, but you are assuming a Gaussian PSF. An actual PSF is most definitely not Gaussian, nor is it constant across the area of the lens (i.e. it changes as you leave the center and approach the corners...do a search for "spot diagram" to see actual lens PSF's produced mathematically from detailed and accurate lens specifications...even for the best of lenses, outside of the most centeral on-axis results, a PSF can be wildly complicated).

I did say that it is not a Gaussian but for many purposes it is close enough. This is not my formula,
it first appeared in a paper on optics and has been used many times since then. If somebody can point out a reference, I will put it there right away.

I also explained why its variance across the frame is not a problem. You just apply the formula in different regions with different values.

I also mentioned there that my point is not that formula but the general principle: multiple blur factors act as a convolution with their convolution. This is a much more universal principle.

Quote
Not to mention the fact that you have to guess the kernel in the first place, so whatever your result, it is immediately affected by what you think the lens is capable of in the first place.

There is no much go guess. The AA filter can very well be approximated with a Gaussian, and the effect of the pixels can just be computed. The only weakness is that you do not really know that the AA filter has strength proportional to the pixel density. But then you have data for many bodies.

But even if you had the absolute lens data, what are you going to do with it? You never answered that question. Let me help you - to see how it performs on a future 5D4 or whatever, you need to make some assumptions on the AA filter, then you need to apply the formula that you do not like. There is no way around that factor.

Quote
Personally, I wouldn't trust any site that provided "lens resolution" results reverse engineered from an image produced by any sensor. I would actually rather take the "camera system" tests than have someone telling me what their best guess is for lens performance.

You probably never go to MRI or CT scan test because instead of just "seeing" what is inside, they "reverse engineer", i.e., they compute it.

Quote
Hmm, DXO's own description on the lens tests page begs to differ:

Quote
DxOMark's comprehensive camera lens test result database allows you to browse and select lenses for comparison based on its characteristics, brand, type, focal range, aperture and price.
This is nitpicking. Try to get the resolution numbers. You cannot get it without choosing a body, and the results are always displayed with the body well visible. Their articles are poorly written but it is not a "rocket science" to realize what you are looking at.

msm

  • PowerShot G16
  • **
  • Posts: 32
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #69 on: July 24, 2013, 02:50:58 PM »
well as I have declare before in many discussion here at CR that  a real MTF test from Hasselblad  MTF lab shows that there are no significant difference between Nikon super tele and Canon, even if Canon are using their fluorite elements compared to Nikons super ED
http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Nikon-AF-S-Nikkor-300mm-and-400mm-f-2.8G-ED-VR-lens-reviews-legendary-performers-in-the-range

and 200/2.0 FROM NIKON, I PERSONALLY RANK THIS LENS AS BETTER THAN 200/1,8 AND 2.0 FROM CANON and so does also http://www.lenstip.com/index.html?test=obiektywu&test_ob=325
http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Nikon-AF-S-Nikkor-200mm-f-2.0G-ED-VR-II-lens-review/Nikkor-AF-S-Nikkor-200mm-f-2G-ED-VR-II-versus-competition


I'll take this page showing real ISO crops over some MTF lab test anyday I'm going to buy a lens to take pictures with:

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=458&Camera=453&Sample=0&FLI=0&API=0&LensComp=648&CameraComp=0&FLIComp=0&APIComp=0

I'll worry about the "real MTF" test next time I buy a lens to take it to the lab.

Mika

  • PowerShot G16
  • **
  • Posts: 45
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #70 on: July 24, 2013, 04:50:48 PM »
I personally think this is the weirdest lens test article I have ever read. It is not a lens test, but a system test.

What it comes to Hasselblad measuring the MTF of the superteles, not all information is included there. Typically reduced secondary spectrum doublets achromats have a better MTF than equal element number apochromatic designs, but the color error is then visible with areas of high contrast difference. I'm also interested in the Hasselblad's methodology of doing the test, could you send me the link, please?

It is not always clear to me whether Nikon is reviewed with software corrections enabled on the color aberrations in these tests or not. For that reason, I'd be a bit suspicious on using current Nikon objectives with film.

Even in the Digital Pictures crops, it is not clear to me, how the crops were obtained from RAWs. But for me it seems that the Canon lens performs actually better than Nikon's at this distance for all tested field points. Whether this difference is significant in real life photography is another story.

By looking at the manufacturers MTFs for 200/2.0s, it seems Canon has optimized the 10 lp/mm performance while slightly sacrificing 30 lp/mm performance compared to Nikon, and averagely the impression seems to be that the Canons 200 mm performs better (to my eye), and considerably better close to the edge. This is nothing new, it is known that images taken with a lens that has a better macro contrast tend to look better than the ones taken with a lens with less contrast but higher micro-contrast. Only in the center crop we can see a bit of the micro-contrast at play, but it isn't clear to me what causes aliasing (or Moire?) on the Canon crop, is it the lens or the sensor.

The MTF tests that we do actually does not include the camera body, but a microscope objective and a known image sensor. It is arguable whether this gives full information since there is further processing due to the software, but it allows checking which lenses are better corrected to begin with.
« Last Edit: July 24, 2013, 05:22:44 PM by Mika »

msm

  • PowerShot G16
  • **
  • Posts: 32
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #71 on: July 24, 2013, 05:41:50 PM »
well as I have declare before in many discussion here at CR that  a real MTF test from Hasselblad  MTF lab shows that there are no significant difference between Nikon super tele and Canon, even if Canon are using their fluorite elements compared to Nikons super ED
http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Nikon-AF-S-Nikkor-300mm-and-400mm-f-2.8G-ED-VR-lens-reviews-legendary-performers-in-the-range

and 200/2.0 FROM NIKON, I PERSONALLY RANK THIS LENS AS BETTER THAN 200/1,8 AND 2.0 FROM CANON and so does also http://www.lenstip.com/index.html?test=obiektywu&test_ob=325
http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Nikon-AF-S-Nikkor-200mm-f-2.0G-ED-VR-II-lens-review/Nikkor-AF-S-Nikkor-200mm-f-2G-ED-VR-II-versus-competition


I'll take this page showing real ISO crops over some MTF lab test anyday I'm going to buy a lens to take pictures with:

http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=458&Camera=453&Sample=0&FLI=0&API=0&LensComp=648&CameraComp=0&FLIComp=0&APIComp=0

I'll worry about the "real MTF" test next time I buy a lens to take it to the lab.


thats no real MTF test of the lens, it is the same way lens tip, photo zone  and others measure the lenses


Never said it was, just tried to make the point that MTF does not tell the entire story and I prefer equipment that produce best images to my eyes and couldn't care less about numbers from a lab. And the Nikon 200f2 looks softer in the corner compared to Canon on those actual images.

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #72 on: July 24, 2013, 07:22:03 PM »
Never said it was, just tried to make the point that MTF does not tell the entire story and I prefer equipment that produce best images to my eyes and couldn't care less about numbers from a lab. And the Nikon 200f2 looks softer in the corner compared to Canon on those actual images.

Not that I disagree with you about the visual evidence vs. the lab results ... but if your goal is to get the whole PSF (the image of an ideal point), then the MTF is giving essentially its Fourier transform, and from there, you get the PSF in a direct and a stable way. This does not work is such a straightforward way when the PSF is too concentrated near a single pixel. Still, the whole MTF is in some sense the "complete data" that DXO is hiding from us.

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #72 on: July 24, 2013, 07:22:03 PM »

msm

  • PowerShot G16
  • **
  • Posts: 32
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #73 on: July 25, 2013, 01:58:01 AM »
Never said it was, just tried to make the point that MTF does not tell the entire story and I prefer equipment that produce best images to my eyes and couldn't care less about numbers from a lab. And the Nikon 200f2 looks softer in the corner compared to Canon on those actual images.

Not that I disagree with you about the visual evidence vs. the lab results ... but if your goal is to get the whole PSF (the image of an ideal point), then the MTF is giving essentially its Fourier transform, and from there, you get the PSF in a direct and a stable way. This does not work is such a straightforward way when the PSF is too concentrated near a single pixel. Still, the whole MTF is in some sense the "complete data" that DXO is hiding from us.

Ages since I had fourier analysis, but as far as I recall that theory is based on point sampling. An image sensor uses area sampling. Has it been shown that fourier analysis still applies?

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #74 on: July 25, 2013, 02:17:22 AM »
Never said it was, just tried to make the point that MTF does not tell the entire story and I prefer equipment that produce best images to my eyes and couldn't care less about numbers from a lab. And the Nikon 200f2 looks softer in the corner compared to Canon on those actual images.

Not that I disagree with you about the visual evidence vs. the lab results ... but if your goal is to get the whole PSF (the image of an ideal point), then the MTF is giving essentially its Fourier transform, and from there, you get the PSF in a direct and a stable way. This does not work is such a straightforward way when the PSF is too concentrated near a single pixel. Still, the whole MTF is in some sense the "complete data" that DXO is hiding from us.

Ages since I had fourier analysis, but as far as I recall that theory is based on point sampling. An image sensor uses area sampling. Has it been shown that fourier analysis still applies?

I think you are talking about something different: the sampling theorem (still connected to Fourier analysis). The sampling theorem still applies to "sampling" by integration over the whole pixel (you made a very good observation) but it needs to be modified. Imagine the whole image first convoluted with a pixel (a function equal to 1 in a square of the size of a pixel, zero otherwise). This convolution is band limited (same limit), if the original image is. Then you point sample that.

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #74 on: July 25, 2013, 02:17:22 AM »