Why so long on the DXOMark and DPReviews?

[psolberg]

I got my 5d III the week before last. I am sure both those organizations got one from at least the first shipped batch. Why is it taking so long to come out with anything other than a preview? Are they running into issues?

I am pretty happy with the 5d3, although the wife is a little mad about the price.

I am really curious about how low the DXOMark score will be and if it is causing them to rethink their test. I just think they might worry it would damage what credibility they have if the nikon 800 had a 95 and the 5d3 had a 85 or so. They clearly are not that far apart unless the Nikon lenses are really holding them back. I don't think that is the case given he zeiss comparisons Lensrentals did.

they would damage their credibility if they adjusted their test to make the 5DmkIII score better 8) so no, that's not the reason. More likely they are just swampped with other things.

 

[well_dunno]

LOL apparently the reason for delayed DxO evaluation of the mark III is because they have not been able to get their hands on a production sample

http://forum.dxomark.com/index.php/topic,720.0.html

 

[Tracy Pinto]

Attacking the integrity of the Dx0 testing labs and then wondering if it is a conspiracy that they are delaying their disreputable work is somewhat funny.

My most important resource is myself. I need to know what I intend to use the new camera/lens for and what I hope to gain by making a new purchase. Then I read away only at sources I trust to sort out what I need to be concerned about in making my trade-offs. As an informed buyer I should not be surprised. All the cameras I have bought had much more features at purchase time than I had skill... and they all had limitations. My 5D Mark III is no exception. I hope to master it and improve as a photographer as I do.

 

[Tcapp]

If I already have *MY* grubby little hands on a 5d3, why doesn't a website like DXO have one yet?! Weird.

 

[JR]

LOL apparently the reason for delayed DxO evaluation of the mark III is because they have not been able to get their hands on a production sample

http://forum.dxomark.com/index.php/topic,720.0.html

This is really odd indeed! Am sure I can get them one quickly! I guess they are not on Canon "preferred" list, if ever such list exist!

 

[wickidwombat]

If I already have *MY* grubby little hands on a 5d3, why doesn't a website like DXO have one yet?! Weird.

Nikon didn't send them one

 

[psolberg]

Attacking the integrity of the Dx0 testing labs and then wondering if it is a conspiracy that they are delaying their disreputable work is somewhat funny.

Potentially questioning DxO's integrity, not attacking it. There was no allegation of a conspiracy. Rather, I noted that there are other means of conveying bias than orienting the tests so that they favor the Nikon. They can do this alone without conspiring with Nikon or third parties. Bottom line: it is a critical analysis, but not one that alleges improper conduct. Please do not misconstrue my words.

I don't think they don't favor a brand per se or orient their results to favor anybody. Their test is always the same and it isn't reformulated on a per camera basis to help or sink a particular body. They simply start with a fixed criteria and methods and carry judgement on those results. Nikon's choices simply score better because they more closely resemble their interpretation of what is important in an imaging sensor. It could have been the case Canon or Sony would have been the best if those companies had chosen different designs and specs on their sensors. DXO simply provides a ruler. If you're not 7 feet tall, changing the ruler doesn't change the fact you're not 7 feet tall.

the bottom line is that no matter who wins, they'll get accused of biased so I see no reason why they should change anything

nothing stops canon from improving the resolution and dynamic range of their bodies and until they don't, they will simply not do very well on these normalized tests. this is entirely canon's fault, not dxo's.

 

[marekjoz]

We're not far from starting a ranking of camera and lenses review sites...

<projected sarcasm quote>
DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"
</projected sarcasm quote>

 

[JR]

We're not far from starting a ranking of camera and lenses review sites...

DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"

Not a bad idea!

 

[SambalOelek]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

 

[well_dunno]

We're not far from starting a ranking of camera and lenses review sites...

DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank"

Not a bad idea!

lol +1!

I think it would be great if Klaus /photozone.de maintained a database and some sort of rating with all the tests he has...

 

[PhilDrinkwater]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

While I largely agree, I also question the ethics of a company which wouldn't take that possibility seriously.

Personally when it comes to sensors, I prefer my eyes to do the viewing because they will ignore the 1/3 to 1/2 stop differences that look quite significant when you look just at numbers. That said, I think they are "interesting". I just wouldn't base any decision on them.

 

[Wrathwilde]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

They said the v2 "offers slightly less resolution" but tests better on chromatic aberration. I fail to see how that qualifies as "significantly worse". Also, "These 2 lenses are very similar in term of transmission, distortion and vignetting", again hardly a condemning review.

The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results.

In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.

 

[marekjoz]

I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results.

In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

 

[Wrathwilde]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

Personally I'd be suspect of any site that cherry picked to find the best possible sample to test, or tested several and only published the best performer. This is the reason that the most reputable sites test units acquired from the retail channel, and not cherry picked units from the manufacture.

I'd also be suspect of any test site that would bow to pressure from end users to redo their tests. Now, if the manufacture claimed that the results were far different than their own in house testing, and contacted them about the possibility that the lens they tested might be defective... and paid for them to acquire another sample, then sure, they should be willing to retest. That said, they are under no obligation to do so.

Sure you can speculate they got a bad copy, suggesting that their credibility is shot because a lens didn't test well is a different matter.

Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available.

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

 

[marekjoz]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available.

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Wrathwilde - yes, I'm aware of that and I'm not saying they cheat. I'd rather say they have too much too loose to manipulate. But it's strange anyway, when they seem to be the only site saying 70-200 2.8 II is worse than mk1. Everybody else is wrong, they are wrong or sth else is wrong? If this lens is considered to be better than mk1, then maybe they tested a bad copy. And it's ok if it indicates how good or bad the Canon internal quality check was. But most probably it doesn't show how these lenses good in general are. And this is the problem. Such a test should present the potential of design, materials, build etc. of a product in general but not of a single copy. That's my opinion.

 

[marekjoz]

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Yes, but people believing in conspiracy say, that as far as Nikon uses their products and pays for commercial, then this absolute independence might be disturbed.
Yes, I believe that fanboys don't believe in independent tests. That's absolutely true. But in this case, anyway I don't find them as the only ones with groundbreaking discovery.

Anyway - I really don't know. I just mention what I'm aware of. I don't have nor had neither 70-200 2.8 mk1 nor mk2 so can't say.

 

[moreorless]

If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

 

[LetTheRightLensIn]

If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

You got that in reverse it is the Sony Exmors where the DR holds up well in the low ISOs and the Nikon and especially Canon ones where the lines curves over and you don't keep gaining the expected stop better each stop you go below ISO800-1600 or so.

 

[moreorless]

If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

You got that in reverse it is the Sony Exmors where the DR holds up well in the low ISOs and the Nikon and especially Canon ones where the lines curves over and you don't keep gaining the expected stop better each stop you go below ISO800-1600 or so.

It doesnt really matter how you state it, my point was that the DxO headline scores will tend to overstate the performance of a sensor who's DR declines in a steady fashion over one who's DR holds steady for longer.

Add to that of course that Sony hasnt produced a FF sensor with the extreme ISO range of the D3s, D4 and 1DX which DxO also do not take into account with their headline scores.

Just seems to me that thus headline marks are rather overused given that they are to some extent subjective, you don't for example see people throwing around photozone marks out of 5 in the same fashion.