Why so long on the DXOMark and DPReviews?

[JR]

LOL apparently the reason for delayed DxO evaluation of the mark III is because they have not been able to get their hands on a production sample 

http://forum.dxomark.com/index.php/topic,720.0.html

This is really odd indeed!  Am sure I can get them one quickly!  I guess they are not on Canon "preferred" list, if ever such list exist!

 

[wickidwombat]

If I already have *MY* grubby little hands on a 5d3, why doesn't a website like DXO have one yet?! Weird.

Nikon didn't send them one

[psolberg]

Attacking the integrity of the Dx0 testing labs and then wondering if it is a conspiracy that they are delaying their disreputable work is somewhat funny.

Potentially questioning DxO's integrity, not attacking it.  There was no allegation of a conspiracy.  Rather, I noted that there are other means of conveying bias than orienting the tests so that they favor the Nikon.  They can do this alone without conspiring with Nikon or third parties.  Bottom line: it is a critical analysis, but not one that alleges improper conduct.  Please do not misconstrue my words.

I don't think they don't favor a brand per se or orient their results to favor anybody. Their test is always the same and it isn't reformulated on a per camera basis to help or sink a particular body. They simply start with a fixed criteria and methods and carry judgement on those results. Nikon's choices simply score better because they more closely resemble their interpretation of what is important in an imaging sensor. It could have been the case Canon or Sony would have been the best if those companies had chosen different designs and specs on their sensors. DXO simply provides a ruler. If you're not 7 feet tall, changing the ruler doesn't change the fact you're not 7 feet tall.

the bottom line is that no matter who wins, they'll get accused of biased so I see no reason why they should change anything 

nothing stops canon from improving the resolution and dynamic range of their bodies and until they don't, they will simply not do very well on these normalized tests. this is entirely canon's fault, not dxo's.

[marekjoz]

We're not far from starting a ranking of camera and lenses review sites...

<projected sarcasm quote>
DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"
</projected sarcasm quote>

[JR]

We're not far from starting a ranking of camera and lenses review sites...

DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"

Not a bad idea!

[SambalOelek]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

[well_dunno]

We're not far from starting a ranking of camera and lenses review sites...

DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"

Not a bad idea!

lol +1!

 I think it would be great if Klaus /photozone.de maintained a database and some sort of rating with all the tests he has...

[scrappydog]

nothing stops canon from improving the resolution and dynamic range of their bodies and until they don't, they will simply not do very well on these normalized tests. this is entirely canon's fault, not dxo's.

+1.  Competition is good.  It fosters innovation.  I just want a camera with good low light performance, excellent latitude, and resolving power.

[PhilDrinkwater]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

While I largely agree, I also question the ethics of a company which wouldn't take that possibility seriously.

Personally when it comes to sensors, I prefer my eyes to do the viewing because they will ignore the 1/3 to 1/2 stop differences that look quite significant when you look just at numbers. That said, I think they are "interesting". I just wouldn't base any decision on them.

[Wrathwilde]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

 

 They said the v2 "offers slightly less resolution" but tests better on chromatic aberration. I fail to see how that qualifies as "significantly worse". Also, "These 2 lenses are very similar in term of transmission, distortion and vignetting", again hardly a condemning review.

  The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

  That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results. 

  In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.

[marekjoz]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

  The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

  That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results. 

  In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

[Wrathwilde]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

  Personally I'd be suspect of any site that cherry picked to find the best possible sample to test, or tested several and only published the best performer. This is the reason that the most reputable sites test units acquired from the retail channel, and not cherry picked units from the manufacture.

  I'd also be suspect of any test site that would bow to pressure from end users to redo their tests. Now, if the manufacture claimed that the results were far different than their own in house testing, and contacted them about the possibility that the lens they tested might be defective... and paid for them to acquire another sample, then sure, they should be willing to retest. That said, they are under no obligation to do so. 

  Sure you can speculate they got a bad copy, suggesting that their credibility is shot because a lens didn't test well is a different matter. 

Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available. 

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

[marekjoz]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

 
Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available. 

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Wrathwilde - yes, I'm aware of that and I'm not saying they cheat. I'd rather say they have too much too loose to manipulate. But it's strange anyway, when they seem to be the only site saying 70-200 2.8 II is worse than mk1. Everybody else is wrong, they are wrong or sth else is wrong? If this lens is considered to be better than mk1, then maybe they tested a bad copy. And it's ok if it indicates how good or bad the Canon internal quality check was. But most probably it doesn't show how these lenses good in general are. And this is the problem. Such a test should present the potential of design, materials, build etc. of a product in general but not of a single copy. That's my opinion.

[marekjoz]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

 
Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Yes, but people believing in conspiracy say, that as far as Nikon uses their products and pays for commercial, then this absolute independence might be disturbed.
Yes, I believe that fanboys don't believe in independent tests. That's absolutely true. But in this case, anyway I don't find them as the only ones with groundbreaking discovery.

Anyway - I really don't know. I just mention what I'm aware of. I don't have nor had neither 70-200 2.8 mk1 nor mk2 so can't say.

[scrappydog]

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.
edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses...

Perhaps your observation is true.  However, I do have common sense.  Check out this link graciously provided by another CR poster: http://www.dxo.com/us/image_quality/customers2. 

To quote DxO's site: "Here is just a sample of our current customers:..." with a picture of Nikon's image below it.  I think it is fair to say that they have ties to camera and lens manufacturers because they are expressly advertising these businesses as their customers on their site.  Paste the link in a browser and look for yourself.  This isn't conjecture nor is it conspiracy theory; it is fact.

EDIT:
I created a PDF of the website.  See the attached.  Opening the page sometimes defaults to their Home page.