DxOMark Splits from DXO Labs

[Canon Rumors Guy]

DxOMark has split from DXO Labs and the latter will now be focused on software development, most notably the recently acquired Nik Software suite, which they say <a href="http://www.canonrumors.com/great-news-dxo-to-release-new-nik-collection-software-in-2018/">will be re-released in 2018</a>.</p>
<p>DxOMark will continue its camera and lens testing and will reportedly become “<a href="http://image-sensors-world.blogspot.com/2018/01/dxomark-separates-from-dxo-labs.html">more objective and less affected by DXO Lab products</a>” according to <a href="http://image-sensors-world.blogspot.com/2018/01/dxomark-separates-from-dxo-labs.html">Image Senors World</a>.</p>


<p>DxOMark does have some credibility issues in certain circles when it comes to testing, so hopefully this will be a good thing and will start changing public perception. We all love objective reviews and those are harder and harder to come by.</p>
<p><strong>The email from DXOMark</strong></p>
<blockquote><p>We’ve had an important internal change as well: In September, DxOMark Image Labs was spun off from DxO Labs. DxOMark Image Labs is now a privately-owned, independent company. As such, we continue to pursue the development and commercialization of image quality solutions and services that support our customers in designing the best-quality camera systems for a range of markets, including smartphones, DSC/DSLRs, drones, action cams, surveillance, and automotive.</p></blockquote>
<p>My head’s spinning a bit, too many “DxOs”, so hopefully I understand this correctly.</p>
<span id="pty_trigger"></span>

 

[ahsanford]

If they...

1) Stop scoring their own product's multi-shot mode and comparing it against competitors with similar multi-shot feature but only report their single shot performance,

2) Abolish their overall score or publish the overall score algorithm in full, so the arbitrary and inconsistent rankings cease,

3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon,

4) Have f/22 performance affect a lens's sharpness rating (see 35L II vs. 35 Art for giggles)

...they have a fighting chance of my taking them seriously.

- A

 

[neuroanatomist]

5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.

 

[3kramd5]

5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.

6. Stop considering camera dynamic range for lens scoring.

7. Start using appropriate (realistic) subject distance in lens testing; putting an ultra wide angle lens 5 feet from a chart is hardly how it will be used, and certainly isn’t what it was designed for.

 

[neuroanatomist]

3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon

6. Stop considering camera dynamic range for lens scoring.

More generally, base lens scores on the optical measurements of the lens itself and not on the combination of camera+lens.

 

[neuroanatomist]

My head’s spinning a bit, too many “DxOs”, so hopefully I understand this correctly.

DxO Labs = DxO PhotoLab (RAW converter), DxO FilmPack, DxO ViewPoint, the Nik suite, and the DxO One camera.

DxOMark Image Labs = DxO Analyzer (B2B industry sensor testing software) and the camera score database (which is basically DxO Analyzer data from production cameras repackaged for public release)

 

[ahsanford]

5) Stop heavily weighting base ISO in the sensor scoring. Based on one assessment comprising analysis of EXIF data for ~20 million images, ISO 100 is used for ~30% of pictures.

Killing off the overall score or breaking Dynamic Range into two (or three) scores...

Landscape, Studio / "Controlled"

General use / "Field"

Low-Light / "Challenging"

...would address this. Seeing as the overall score is the metacritic/RT reveal for the internet masses to frenzy over -- and it must drive page hits -- I just don't think the former will happen.

Also, they could do this ISO dependent output with the noise and color depth metrics as well.

- A

 

[3kramd5]

3) Stop taking the exact same lens from Sigma or Zeiss and declaring it disappointing on Canon and mind-blowing on Nikon

6. Stop considering camera dynamic range for lens scoring.

More generally, base lens scores on the optical measurements of the lens itself and not on the combination of camera+lens.

There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

A transfer function doesn't tell the whole story. For example when I was still using an A7Rii I often adapted Canon lenses. However, Canon lenses aren't designed for the unique filter stack in front of the Sony camera sensor, and perform better on the native platform than adapted, a fact which would not turn up on an optical bench.

As a tool to help determine optical performance of lenses you're interested in on bodies you're interested in, their numbers are useful. It would be better if they stopped scoring lenses with different mounts against each other. Best would be to abandon the composite scoring scheme, but that is probably their biggest web traffic generation mechanism.

 

[ahsanford]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

 

[neuroanatomist]

There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Except that DxOMark does report sensor scores independent of lenses (or rather, without regard to lens). But their DxOMark Lens Score is camera-dependent. Moreover, they draw conclusions such as 'this brand's 24-70/2.8 zoom is better than that brand's equivalent' based on the sensor contribution, when the lens' optics alone would lead to the opposite conclusion.

Then there's:

8. Stop assuming that we all use our cameras and lenses in a dimly lit warehouse.

Lens scores are based on 'performance in lighting of 150 lux' which is why, according to the DXOMark lens score, the 50/1.8 is a better optic than the 300/2.8 or 600/4.

 

[3kramd5]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

 

[ahsanford]

8. Stop assuming that we all use our cameras and lenses in a dimly lit warehouse.

Lens scores are based on 'performance in lighting of 150 lux' which is why, according to the DXOMark lens score, the 50/1.8 is a better optic than the 300/2.8 or 600/4.

Also:

9. When the company you constantly batter for too having few pixels behind the lens finally puts out a sensor that outresolves your preferred gear maker, stop rewriting your unpublished overall lens score to still find your preferred gear maker makes better stuff. You can't scream 'but the resolution on this D800E is amazing' for a couple years and then start scream 'no no, hear me out: transmission is the new resolution, yo' the minute the 5DS R is on the scene. :

- A

 

[3kramd5]

There is validity to the pairing. Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Except that DxOMark does report sensor scores independent of lenses (or rather, without regard to lens). But their DxOMark Lens Score is camera-dependent. Moreover, they draw conclusions such as 'this brand's 24-70/2.8 zoom is better than that brand's equivalent' based on the sensor contribution, when the lens' optics alone would lead to the opposite conclusion.

And changing both of those things would improve their credibility.

 

[candc]

Their measurements can be useful but the conglomerate score they assign can be misleading for sure.

I use dxo software and I really like the new local adjustment tools in "photolab"

 

[neuroanatomist]

And changing both of those things would improve their credibility.

I agree, although to be honest their credibility with photographers is probably rather irrelevant given that their primary source of income is B2B.

 

[Valvebounce]

Hi Folks.
I’m not certain, but I don’t think I have ever used DxO lens or camera scores, but I do use DxO Optics Pro and Photolab, my concern is whether the software side will still have access to the excellent lens correction data?
This question is based on the assumption that the data for the lens corrections comes from the lens score tests, if this is wrong I have just reaffirmed the “to ass-u-me” saying! ;D

Cheers, Graham.

 

[RogerCicala]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

 

[3kramd5]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

Agreed, however

1) some of those variables are pertinent and may be unwise to eliminate (for example if an 85mm lens from brand A has fewer losses than an 85mm lens from brand B, it matters little to someone who can not use brand B lenses. Similarly, if you want an 85mm lens, focal length is a variable which should not be eliminated, whereas if you’re after the most perfect optics and will work around focal length, it can be), and
2) not all scientific data is readily usable. Your work is great and very interesting; I read your blogs including bench level lens testing. However I’ve yet to come up with a way to use an MTF chart.

In this particular instance, the variable I question eliminating is the camera sensor's optical stack.

I think we can agree that

A) the MTF of an optical system is a function of the MTF of the constituent members. That is true even when the optical system is a camera lens: each element has a transfer function, which contribute to the lens transfer function as a whole. Some elements are included to correct the affects of others.

B) since the advent of digital ILCs, lenses have been designed for the cameras to which they'll be mounted, ergo the sensor and its optical stack (as opposed to lenses designed for film cameras).

Some third party lenses may be sensor agnostic, but they probably should not be. Interviews with both Sigma and Laowa/Venus refer to the importance of designing for the optical stack. It's part of the path, indeed it's the final element before the sensor. Putting a lens on an optical bench without that stack to eliminate the variable eliminates the proper environment for the lens, similar to (though less impactful than) pulling the rear element off a camera lens.

Occam's razor and all, maybe I'm just dense. However, it seems to me that by isolating the variables to [camera lens], one introduces the risk that the remaining variable is being used inappropriately.

 

[ahsanford]

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

I'm sorry, let's just take a moment and reflect that I just summoned Uncle Rog with Bat-symbol like post. Kind of proud of myself at the moment.

Thanks, Uncle Rog. We love what you do, especially the bit about transparency of methods, not coming up with some BS unpublished aggregate score and (most of all) being a wonderful resource with a name that we can ask questions to. Please travel to France (or wherever Le DXO Scoremaster Generale is located in his/her secret bunker) and straighten them out.

- A

 

[RogerCicala]

Much like nobody outside a semiconductor OEM is likely to have good reason to test a sensor without a lens, nobody is going to use a camera lens without a camera (unlike say a microscope or binoculars).

Obi Rog Kenobi at LR pets his beloved OLAF... and feels a powerful disturbance in the force.

- A

Right. There he is testing it, but not using it. I betcha when Roger uses his camera lenses, they’re mounted to cameras

Granted you have to have to use it to test it, but the distinction is clear enough.

Science, as I was trained to do it, is about eliminating as many variables as possible to test one variable in a reasonably large sample size. Hence, we test just the lenses and a reasonably large sample size of the lenses.

Carefully measuring one third grader and then stating that all third graders are 67.325 secret measurements tall is bad science.

Agreed, however

1) some of those variables are pertinent and may be unwise to eliminate (for example if an 85mm lens from brand A has fewer losses than an 85mm lens from brand B, it matters little to someone who can not use brand B lenses. Similarly, if you want an 85mm lens, focal length is a variable which should not be eliminated, whereas if you’re after the most perfect optics and will work around focal length, it can be), and
2) not all scientific data is readily usable. Your work is great and very interesting; I read your blogs including bench level lens testing. However I’ve yet to come up with a way to use an MTF chart.

In this particular instance, the variable I question eliminating is the camera sensor's optical stack.

I think we can agree that

A) the MTF of an optical system is a function of the MTF of the constituent members. That is true even when the optical system is a camera lens: each element has a transfer function, which contribute to the lens transfer function as a whole. Some elements are included to correct the affects of others.

B) since the advent of digital ILCs, lenses have been designed for the cameras to which they'll be mounted, ergo the sensor and its optical stack (as opposed to lenses designed for film cameras).

Some third party lenses may be sensor agnostic, but they probably should not be. Interviews with both Sigma and Laowa/Venus refer to the importance of designing for the optical stack. It's part of the path, indeed it's the final element before the sensor. Putting a lens on an optical bench without that stack to eliminate the variable eliminates the proper environment for the lens, similar to (though less impactful than) pulling the rear element off a camera lens.

Occam's razor and all, maybe I'm just dense. However, it seems to me that by isolating the variables to [camera lens], one introduces the risk that the remaining variable is being used inappropriately.

You do realize that it was our work on sensor stack thickness a few years ago that started all the public discussion on stack thickness, and that we place the appropriate sensor stack in the optical path for each lens tested? We do this for the tests you read, and for the ones you don't read that we do for various manufacturers (who, BTW, always approve the stack thickness we're testing their lenses at).