DxOMark Splits from DXO Labs

[3kramd5]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

Agreed, however

1) some of those variables are pertinent and may be unwise to eliminate (for example if an 85mm lens from brand A has fewer losses than an 85mm lens from brand B, it matters little to someone who can not use brand B lenses. Similarly, if you want an 85mm lens, focal length is a variable which should not be eliminated, whereas if you’re after the most perfect optics and will work around focal length, it can be), and
2) not all scientific data is readily usable. Your work is great and very interesting; I read your blogs including bench level lens testing. However I’ve yet to come up with a way to use an MTF chart.

In this particular instance, the variable I question eliminating is the camera sensor's optical stack.

I think we can agree that

A) the MTF of an optical system is a function of the MTF of the constituent members. That is true even when the optical system is a camera lens: each element has a transfer function, which contribute to the lens transfer function as a whole. Some elements are included to correct the affects of others.

B) since the advent of digital ILCs, lenses have been designed for the cameras to which they'll be mounted, ergo the sensor and its optical stack (as opposed to lenses designed for film cameras).

Some third party lenses may be sensor agnostic, but they probably should not be. Interviews with both Sigma and Laowa/Venus refer to the importance of designing for the optical stack. It's part of the path, indeed it's the final element before the sensor. Putting a lens on an optical bench without that stack to eliminate the variable eliminates the proper environment for the lens, similar to (though less impactful than) pulling the rear element off a camera lens.

Occam's razor and all, maybe I'm just dense. However, it seems to me that by isolating the variables to [camera lens], one introduces the risk that the remaining variable is being used inappropriately.

You do realize ... that we place the appropriate sensor stack in the optical path for each lens tested?

Clearly I did not. I did read in a blog (I believe) about optical glass being added for a given test but did not know it is done as a matter of course.

Kudos. My above post is redundant.

 

[jeffa4444]

3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon

6. Stop considering camera dynamic range for lens scoring.

More generally, base lens scores on the optical measurements of the lens itself and not on the combination of camera+lens.

There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

A transfer function doesn't tell the whole story. For example when I was still using an A7Rii I often adapted Canon lenses. However, Canon lenses aren't designed for the unique filter stack in front of the Sony camera sensor, and perform better on the native platform than adapted, a fact which would not turn up on an optical bench.

As a tool to help determine optical performance of lenses you're interested in on bodies you're interested in, their numbers are useful. It would be better if they stopped scoring lenses with different mounts against each other. Best would be to abandon the composite scoring scheme, but that is probably their biggest web traffic generation mechanism.

What DXO doesn't do and If I'm reading your post correctly you don't seem to understand is all reputable companies would a. test the camera sensor excluding the lens b. test the lens (can be two part of three part i.e. test on MTF, test on projector, test on f stop / t stop machine) c. test the lens on the camera its to be used with.
The lens tells you nothing about whether the camera dynamic range is accurate or not or a host of other electronic measurements about its performance. Likewise all the top lens manufactures Ive had the pleasure to visit or indeed the company I work for all primarily want to see how a lens projects as much as any MTF reading its far more simple than shooting on the camera to see vingetting at different stops or in the case of zooms at different focal lengths. Its also easier to see colour fringing & aberrations. The MTF will also tell you with multiple off-centre readings if an element is out of alignment.

 

[3kramd5]

Hi Jeff, I don’t think you read my post correctly. Of course OEM’s independently test components. It’s at the end used level where I’m questioning the value of isolated data.

 

[Bdube]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

Agreed, however

1) some of those variables are pertinent and may be unwise to eliminate (for example if an 85mm lens from brand A has fewer losses than an 85mm lens from brand B, it matters little to someone who can not use brand B lenses. Similarly, if you want an 85mm lens, focal length is a variable which should not be eliminated, whereas if you’re after the most perfect optics and will work around focal length, it can be), and
2) not all scientific data is readily usable. Your work is great and very interesting; I read your blogs including bench level lens testing. However I’ve yet to come up with a way to use an MTF chart.

In this particular instance, the variable I question eliminating is the camera sensor's optical stack.

I think we can agree that

A) the MTF of an optical system is a function of the MTF of the constituent members. That is true even when the optical system is a camera lens: each element has a transfer function, which contribute to the lens transfer function as a whole. Some elements are included to correct the affects of others.

B) since the advent of digital ILCs, lenses have been designed for the cameras to which they'll be mounted, ergo the sensor and its optical stack (as opposed to lenses designed for film cameras).

Some third party lenses may be sensor agnostic, but they probably should not be. Interviews with both Sigma and Laowa/Venus refer to the importance of designing for the optical stack. It's part of the path, indeed it's the final element before the sensor. Putting a lens on an optical bench without that stack to eliminate the variable eliminates the proper environment for the lens, similar to (though less impactful than) pulling the rear element off a camera lens.

Occam's razor and all, maybe I'm just dense. However, it seems to me that by isolating the variables to [camera lens], one introduces the risk that the remaining variable is being used inappropriately.

Ah, this is a great misunderstanding. Roger has already handled the whole sensor stack thing.

However, it is certainly untrue that the individual elements have transfer functions, that, when multiplied, produce the system transfer function. This is an area where optics departs from ECE, or general LSI theory.

Imagine you have a 1 inch diameter, 2 inch focal length lens which is diffraction limited. This gives you an MTF described by an arc cosine formula you are free to to find online. Consider that you get another, and place it in contact with the previous one. You now have a 1 inch diameter, 1 inch focal length doublet. Since each piece was diffraction limited, the result is also diffraction limited and will have diffraction limited f/1 performance. Not the square of diffraction limited f/2 performance, which would be quite substantially worse and incorrect.

FWIW, the UVIR cut filter and OLPF are under the coverglass. It is not the last thing before the sensor.

I recommend reading Intro to Fourier Optics by Goodman, or Principles of Optics by Born & Wolf, before writing about transfer functions and LSI theory in optics.