Days since DXO sensor score of 5DS = 112. Number of lenses re-tested = 0.

[chromophore]

Re: Days since DXO sensor score of 5DS = 70. Number of lenses re-tested = 0.

If other photographers didn't put so much stock in DxO's random number generator, then I wouldn't care that they have the biases that they do. If other photo blogs and forums didn't excitedly announce sensor and lens "scores" each time a new product is released and "tested," then it wouldn't matter what they say.

But DxO promotes itself as some kind of "objective" testing organization, and makes outrageously false claims even when faced with abundant evidence to the contrary. They make money off of deliberate lies and deception. They make a mockery of scientific and statistical methodology by pretending to be scientific.

There are comparisons that we know are true because we can see it with our own eyes. And then there's DxO BS and bias and hypocrisy. Even KR doesn't put himself out there as if he's making objective assessments.

It's not that I care if Sony or Nikon fans use DxO scores to justify the superiority of their camera systems of choice. Conversely, I don't care if Canon fans bash DxO for the same reason. As a scientist, what I care about is that anybody even believes that what DxO does is scientific, because that to me is not a reflection on Sony, Nikon, Canon, or any other camera/lens manufacturer, but on science itself, and on the way people who lack the ability to think critically about data are so easily misled by the trappings of DxO's pseudoscientific babble.

 

[zim]

Re: DXO replies regarding Metamerism score difference to prove

Some one on dpr followed up DPR about their numbers. DXO has a trademark for generating worthless numbers.

http://www.dpreview.com/forums/post/56459976

Of course. I certainly do not take them seriously, but their ranking system is outright inflammatory and I'd like them to simply follow their procedure, re-test one 'wretched' Canon lens in front of a 50 MP sensor and watch it rocket up their charts.

Please don't mistake this for fanboyism, I don't need to see that my 70-200 f/2.8L IS II isn't the 732nd best lens any longer. I just want to see a broken rating system deliver a wildly different result on the same lens.

- A

Yes but option 3 is well under way so it's just not going to happen.

Anyone ever tried emailing them and asking! ;D

 

[ahsanford]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

86 days since the 5DS/5DS R sensor scores. Zero lenses re-tested.

Even the A7R II has been out for a while. One wonders if DXO won't re-test any lenses until SoNikon finally have a greater than 50 MP rig. :

- A

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

86 days since the 5DS/5DS R sensor scores. Zero lenses re-tested.

Even the A7R II has been out for a while. One wonders if DXO won't re-test any lenses until SoNikon finally have a greater than 50 MP rig. :

- A

I wonder if they are just done re-testing lenses every time a new body comes out.

With Sony releasing uncompressed 14-bit RAW firmware this month for the A7Rii, they were asked when they'll retest. They said "Unfortunately, we do not retest gear, since it would be too time-consuming, and because of other priorities on the road map."

Perhaps that extends to lenses (in particular strapping them to every compatible body), which would likely take far more time than merely retesting a single body.

Prediction: only new canon EF lenses will be tested on the 5DS/R, and only new Nikon FX lenses will be tested on whatever replaces the D810.

 

[ahsanford]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

86 days since the 5DS/5DS R sensor scores. Zero lenses re-tested.

Even the A7R II has been out for a while. One wonders if DXO won't re-test any lenses until SoNikon finally have a greater than 50 MP rig. :

- A

I wonder if they are just done re-testing lenses every time a new body comes out.

With Sony releasing uncompressed 14-bit RAW firmware this month for the A7Rii, they were asked when they'll retest. They said "Unfortunately, we do not retest gear, since it would be too time-consuming, and because of other priorities on the road map."

Perhaps that extends to lenses (in particular strapping them to every compatible body), which would likely take far more time than merely retesting a single body.

Prediction: only new canon EF lenses will be tested on the 5DS/R, and only new Nikon FX lenses will be tested on whatever replaces the D810.

Thoughts:

1) http://www.dxomark.com/Cameras/Canon/EOS-7D-Mark-II---Lenses-tested --> There are a boatload of pre-7D2 lenses that were re-tested on it after it came out.

2) DXO can't reasonably hold the high ground that sensors matter with lenses and offer lens + sensor combinations like they have and then suddenly withhold that information. They have always been the brute force 'do it all' review house, so saying 'nah, it will be hard' is a little hollow coming from them.

3) LensTip and PhotoZone have (to my knowledge) both comitted to retesting Canon glass on the 5DS rigs. DXO surely dwarfs them for size / capabilities, but one would think that if the smaller sites are going the extra distance, why shouldn't the big boys?

4) I hear you on the 14 bit comment, but perhaps they meant that they don't retest sensors after they do a sensor review. It is pretty rare to see a sensor get a quality upgrade after it's launched like this.

Just my opinion, of course. You raise fair points.

- A

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Thoughts:

1) http://www.dxomark.com/Cameras/Canon/EOS-7D-Mark-II---Lenses-tested --> There are a boatload of pre-7D2 lenses that were re-tested on it after it came out.

Are there any cameras newer than the 7D2 that have had the lens treatment? No 5DS/R, no A7Rii. Maybe it's new policy. I hope not.

2) DXO can't reasonably hold the high ground

Just stop right there

4) I hear you on the 14 bit comment, but perhaps they meant that they don't retest sensors after they do a sensor review. It is pretty rare to see a sensor get a quality upgrade after it's launched like this.

Certainly, and that statement was specifically made in reference to firmware changing camera output. However, the stated reasoning is time, and given the oodles and oodles of EF lenses, including Canon, Sigma, Tamron, Zeiss, etc., it stands to reason that re-testing them all for each new body may be a relic.

I dunno.

 

[Loswr]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

DxO Optics Pro corrects for lens softness, CA, vignetting, etc., with the corrections based on empirical data. When a new camera comes out, they generate those empirical data by testing that new camera with all the supported lenses. So, I'd argue that the tests are done and the data are available. What may be lacking is time to format all those data for display in DxOMark...if not time, motivation?

 

[ahsanford]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

2) DXO can't reasonably hold the high ground

Just stop right there

Ha! Nice.

You raise fair points about re-testing everything being an untenable position.

PhotoZone and LensTip are choking on that right now. PZ got their hands on a 5DS R but have only tested two lenses so far, and to my knowledge, LT hasn't posted a single lens review on the 50 MP rigs yet.

Perhaps they only retest more popular, staple lenses of the last 5 years, but they have to do something. How will we compare two opposing lenses that came out on either side of a migration to a new test sensor? I want to compare the 35L II to the Sigma 35 Art, or a new 'III' version against the 'II' lens it replaces, which might be 5-10 years apart. I think they have to try to do this... somehow.

If nothing else, the 'III' to 'II' example above helps Canon sell lenses and maintain price. Perhaps Canon and Nikon themselves might loan these lenses out and encourage retesting as a partnership to keep interest in new bodies and lenses as high as possible.

- A

 

[StudentOfLight]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Take a look at the time period between D810 sensor testing and publication of D810 lens recommendations. The lens database update took a long time.

 

[ahsanford]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

DxO Optics Pro corrects for lens softness, CA, vignetting, etc., with the corrections based on empirical data. When a new camera comes out, they generate those empirical data by testing that new camera with all the supported lenses. So, I'd argue that the tests are done and the data are available. What may be lacking is time to format all those data for display in DxOMark...if not time, motivation?

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

So, yes, Neuro, you are spot on. Why haven't they reported this information?

- A

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

DxO Optics Pro corrects for lens softness, CA, vignetting, etc., with the corrections based on empirical data. When a new camera comes out, they generate those empirical data by testing that new camera with all the supported lenses. So, I'd argue that the tests are done and the data are available. What may be lacking is time to format all those data for display in DxOMark...if not time, motivation?

Or the MARK side of the organization is separate from the software side? I kinda doubt it, but you never know.

I always thought the MARK was just a conglomeration of the data they generated in support of their software package, but it seems to have taken on the primary role at DxO, with testing being a selling point in and of itself. Maybe now the DxO One will be kingpin in the hierarchy - they can't resist popping it up every time you browse to a DxO domain...

It's all very strange.

 

[ahsanford]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Take a look at the time period between D810 sensor testing and publication of D810 lens recommendations. The lens database update took a long time.

I noticed that, but I though the 'recommendations' list was more of a roll-up of all the lens testing when it was done. (That takes ages, I'm sure.)

Does DXO post individual dates on when a specific lens is tested on a specific body? I can't seem to find that in their lens reviews.

- A

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Does DXO post individual dates on when a specific lens is tested on a specific body? I can't seem to find that in their lens reviews.

- A

I looked as well. Could be in the source code but I didn't bother checking.

Maybe rather than release data in piecemeal (i.e. on a lens-for-lens basis) they load it all at once (i.e. on a camera basis).

 

[raptor3x]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

Do we actually know this for sure? I can't see any reason they couldn't interpolate corrections from one body to another.

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

Do we actually know this for sure? I can't see any reason they couldn't interpolate corrections from one body to another.

Given how they come up with "print size," I wouldn't be surprised if they use a math model (perhaps using linear inter- or extrapolation). But that would take almost no time at all, so again: where's the beef?

 

[StudentOfLight]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

Do we actually know this for sure? I can't see any reason they couldn't interpolate corrections from one body to another.

Given how they come up with "print size," I wouldn't be surprised if they use a math model (perhaps using linear inter- or extrapolation). But that would take almost no time at all, so again: where's the beef?

Without measurements the strength of the AA filter effect is unknown.

e.g. The 5D-III and 1Ds-III have quite similar resolution but the 5D-III produces a sharper image due to the weaker AA filter. Similarly, comparing 24MP Nikon D3X and the D750, the D750 produces sharper images.

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

Do we actually know this for sure? I can't see any reason they couldn't interpolate corrections from one body to another.

Given how they come up with "print size," I wouldn't be surprised if they use a math model (perhaps using linear inter- or extrapolation). But that would take almost no time at all, so again: where's the beef?

Without measurements the strength of the AA filter effect is unknown.

e.g. The 5D-III and 1Ds-III have quite similar resolution but the 5D-III produces a sharper image due to the weaker AA filter. Similarly, comparing 24MP Nikon D3X and the D750, the D750 produces sharper images.

But if you characterize the effect of AA when testing a body and relate it via some coefficient in your math model, you can wave your hands and say "here's the number" when publishing lens data.

 

[Loswr]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

...you can wave your hands and say "here's the number" when publishing lens data.

That's probably the most accurate description to date of DxOMark's testing methodology.

 

[raptor3x]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Yep. This comment came up earlier: if it's in the software, they've tested the lens on that body.

Do we actually know this for sure? I can't see any reason they couldn't interpolate corrections from one body to another.

Given how they come up with "print size," I wouldn't be surprised if they use a math model (perhaps using linear inter- or extrapolation). But that would take almost no time at all, so again: where's the beef?

Without measurements the strength of the AA filter effect is unknown.

e.g. The 5D-III and 1Ds-III have quite similar resolution but the 5D-III produces a sharper image due to the weaker AA filter. Similarly, comparing 24MP Nikon D3X and the D750, the D750 produces sharper images.

But if you characterize the effect of AA when testing a body and relate it via some coefficient in your math model, you can wave your hands and say "here's the number" when publishing lens data.

I just checked and there are a handful of lens/body combinations that are available in the DxO Optics Module database that are marked as no measurements available in the DxO lens score database. I think that it's very likely that the lens module are generated by interpolated data and honestly I can't see any good reason why that wouldn't be a perfectly reasonable approach.

 

[3kramd5]

Re: Days since DXO sensor score of 5DS = 86. Number of lenses re-tested = 0.

Why do people care about the number of re-tested lenses?

With the number of insults hurled in the direction of DxO it is surprising to see that people care about this.

It's because they marry the data to bodies. If they tested them on an optical bench re-testing would be irrelevant.