July 30, 2014, 09:52:21 AM

Author Topic: Why so long on the DXOMark and DPReviews?  (Read 8301 times)

SambalOelek

  • PowerShot G1 X II
  • ***
  • Posts: 53
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #30 on: April 12, 2012, 08:15:34 AM »
I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.
1D X, 1D IV, 5D III, 6D, 7D, M. Assorted lenses.

canon rumors FORUM

Re: Why so long on the DXOMark and DPReviews?
« Reply #30 on: April 12, 2012, 08:15:34 AM »

well_dunno

  • 7D
  • *****
  • Posts: 356
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #31 on: April 12, 2012, 08:17:26 AM »
We're not far from starting a ranking of camera and lenses review sites...

DXO tests description: "they provide tests where Canon doesn't win because of criteria mostly based on sensor and dynamic range. DXO can blame themselves for being so low in our rank" :)

Not a bad idea!

lol +1!

 I think it would be great if Klaus /photozone.de maintained a database and some sort of rating with all the tests he has...

PhilDrinkwater

  • Guest
Re: Why so long on the DXOMark and DPReviews?
« Reply #32 on: April 12, 2012, 10:25:47 AM »
I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.
While I largely agree, I also question the ethics of a company which wouldn't take that possibility seriously.

Personally when it comes to sensors, I prefer my eyes to do the viewing because they will ignore the 1/3 to 1/2 stop differences that look quite significant when you look just at numbers. That said, I think they are "interesting". I just wouldn't base any decision on them.

Wrathwilde

  • Rebel T5i
  • ****
  • Posts: 141
    • View Profile
    • Anarchy Photography
Re: Why so long on the DXOMark and DPReviews?
« Reply #33 on: April 12, 2012, 10:55:14 AM »
I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.
 

 They said the v2 "offers slightly less resolution" but tests better on chromatic aberration. I fail to see how that qualifies as "significantly worse". Also, "These 2 lenses are very similar in term of transmission, distortion and vignetting", again hardly a condemning review.

  The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

  That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results. 

  In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.
« Last Edit: April 12, 2012, 11:09:35 AM by Wrathwilde »

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: Why so long on the DXOMark and DPReviews?
« Reply #34 on: April 12, 2012, 11:11:05 AM »
I lost quite a bit of faith in DXOMark after they tested the 70-200 f/2.8 IS II and found it to be significantly worse than the v1 in terms of resolution, and refused the possibility of a subaverage sample.

Their sensor tests may still be good, though.

  The tests are what they are, and the results speak for themselves. Just because you're not happy with how the test turned out, or think they may have got a sub-average sample is irrelevant. The test was done with a final production sample. They put the sensor or lens through their specific testing procedures, and those are the results they got... end of story. The v1 production unit they tested was better than the v2. Sure it was probably a fluke, but if you were upgrading, there is also the possibility that your v2 could score worse than your v1. Production samples vary, it's a fact of life.

  That said, I think Nikon tests better because Nikon actually uses DXOMark equipment to test their sensors, and probably uses that data to determine what's working and what's not in terms of DR, color depth, and ISO. Not that they tune their sensors to beat the test, but use the test's to determine which of their sensor designs should produce the best real world results. 

  In no way do I think DxO favors one brand over another, the fact that Nikon is a customer doesn't influence their DxO scores, as the scores aren't a matter of subjective bias. The scores are based on a rigid set of criteria. Nikon scores well because they use the same equipment and methods to test their sensors... then use that data to improve their sensors.

I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

Wrathwilde

  • Rebel T5i
  • ****
  • Posts: 141
    • View Profile
    • Anarchy Photography
Re: Why so long on the DXOMark and DPReviews?
« Reply #35 on: April 12, 2012, 02:49:23 PM »
I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

  Personally I'd be suspect of any site that cherry picked to find the best possible sample to test, or tested several and only published the best performer. This is the reason that the most reputable sites test units acquired from the retail channel, and not cherry picked units from the manufacture.

  I'd also be suspect of any test site that would bow to pressure from end users to redo their tests. Now, if the manufacture claimed that the results were far different than their own in house testing, and contacted them about the possibility that the lens they tested might be defective... and paid for them to acquire another sample, then sure, they should be willing to retest. That said, they are under no obligation to do so. 

  Sure you can speculate they got a bad copy, suggesting that their credibility is shot because a lens didn't test well is a different matter. 

Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available. 

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.
« Last Edit: April 12, 2012, 03:03:26 PM by Wrathwilde »

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: Why so long on the DXOMark and DPReviews?
« Reply #36 on: April 12, 2012, 03:17:49 PM »
I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

 
Lastly, no, they are not sponsored by any Camera or Lens company

From their website -

We test commercial products: in other words, we buy or rent lenses and cameras from the very same retailers that consumers use. When we do test pre-production models (when commercial products are not yet available), we clearly indicate this on our site, and we retest those models when they become commercially available. 

Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Wrathwilde - yes, I'm aware of that and I'm not saying they cheat. I'd rather say they have too much too loose to manipulate. But it's strange anyway, when they seem to be the only site saying 70-200 2.8 II is worse than mk1. Everybody else is wrong, they are wrong or sth else is wrong? If this lens is considered to be better than mk1, then maybe they tested a bad copy. And it's ok if it indicates how good or bad the Canon internal quality check was. But most probably it doesn't show how these lenses good in general are. And this is the problem. Such a test should present the potential of design, materials, build etc. of a product in general but not of a single copy. That's my opinion.
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

canon rumors FORUM

Re: Why so long on the DXOMark and DPReviews?
« Reply #36 on: April 12, 2012, 03:17:49 PM »

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: Why so long on the DXOMark and DPReviews?
« Reply #37 on: April 12, 2012, 03:26:45 PM »
I defended DXO some time ago just because they publish their testing procedures first and then follow them when testing. But I think you will also admit, that when there is a place for some speculation it will always happen. When only one copy (even if production one) was tested, then someone always can ask questions:
1. Was it the only copy they had?
2. If no didn't they "carefully" selected one "special" copy for testing purposes?
3. If they publish tests of only one copy of gear, doesn't it open space for such speculations as above?
4. Do they have any interest in such behaviour if they are sponsored by one producent?

I don't say they do it like described above. But as far as there is a reason to speculate, people will ask questions and doubt.

 
Finally, DxOMark has no ties to or interests of any sort with camera or lens manufacturers, which means that we are completely independent from them.

edit - That and DxOMark has probably figured out that 99.9% of people posting comments have absolutely zero credibility or experience in testing sensors and/or lenses, and are just butt-hurt that the Camera/Lens that they purchased (or planned to purchase) didn't score as high as they believed it should. That and the posters just can't accept that their beloved product isn't the greatest thing since sliced sex on buttered buns.

Yes, but people believing in conspiracy say, that as far as Nikon uses their products and pays for commercial, then this absolute independence might be disturbed.
Yes, I believe that fanboys don't believe in independent tests. That's absolutely true. But in this case, anyway I don't find them as the only ones with groundbreaking discovery.

Anyway - I really don't know. I just mention what I'm aware of. I don't have nor had neither 70-200 2.8 mk1 nor mk2 so can't say.
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

moreorless

  • 6D
  • *****
  • Posts: 579
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #38 on: April 13, 2012, 02:41:45 AM »
If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3298
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #39 on: April 13, 2012, 03:04:33 AM »
If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

You got that in reverse it is the Sony Exmors where the DR holds up well in the low ISOs and the Nikon and especially Canon ones where the lines curves over and you don't keep gaining the expected stop better each stop you go below ISO800-1600 or so.

moreorless

  • 6D
  • *****
  • Posts: 579
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #40 on: April 13, 2012, 03:28:56 AM »
If anyone is looking to "make sensors for DxO tests" then I'd say its Sony rather than Nikon, the D4 with Nikons own sensor in afterall follows Canon performance more closely with DR holding up well thoughout the lower ISO settings.

The D800 on the other hand follows the pattern of Sony's recent sensors with the DR declining steadly below ISO 100 which with the DxO criteria does tend to overplay its performance at bit.

You got that in reverse it is the Sony Exmors where the DR holds up well in the low ISOs and the Nikon and especially Canon ones where the lines curves over and you don't keep gaining the expected stop better each stop you go below ISO800-1600 or so.

It doesnt really matter how you state it, my point was that the DxO headline scores will tend to overstate the performance of a sensor who's DR declines in a steady fashion over one who's DR holds steady for longer.

Add to that of course that Sony hasnt produced a FF sensor with the extreme ISO range of the D3s, D4 and 1DX which DxO also do not take into account with their headline scores.

Just seems to me that thus headline marks are rather overused given that they are to some extent subjective, you don't for example see people throwing around photozone marks out of 5 in the same fashion.

Wrathwilde

  • Rebel T5i
  • ****
  • Posts: 141
    • View Profile
    • Anarchy Photography
Re: Why so long on the DXOMark and DPReviews?
« Reply #41 on: April 13, 2012, 11:41:41 AM »

To quote DxO's site: "Here is just a sample of our current customers:..." with a picture of Nikon's image below it.  I think it is fair to say that they have ties to camera and lens manufacturers because they are expressly advertising these businesses as their customers on their site.  Paste the link in a browser and look for yourself.  This isn't conjecture nor is it conspiracy theory; it is fact.

EDIT:
I created a PDF of the website.  See the attached.  Opening the page sometimes defaults to their Home page.

  Wrong. It means that several large companies find enough value in DxOMark products and services that they are willing to purchase them, and Nikon happens to be one of many. It's not advertising for Nikon, it's advertising for DxOMark. A lot of large companies list who their customers are, so what? It provides a reassurance for other companies who are looking at purchasing their equipment and services, and therefore further establishes their legitimacy. It doesn't make DxOMark beholden to Nikon in any way.

  A customer is a customer, either they find value in your products and services, or they don't. Nikon currently finds that DxoMark's equipment and procedures are helpful in their quest to make better sensors. If anything this makes Nikon beholden to DxOMark, not the other way around.

PhilDrinkwater

  • Guest
Re: Why so long on the DXOMark and DPReviews?
« Reply #42 on: April 13, 2012, 11:47:27 AM »
It doesn't make DxOMark beholden to Nikon in any way.
No. But it doesn't discount it either.

I doubt there's anything in the DxO / Nikon "connection" but to assume it is or isn't true (ie. to assume either side is true) is as random as taking the opposite stance.

The simple truth is this:
* Nikon pay DxO for "something".
* We don't know what it is.
* We don't know whether it gives Nikon any advantage.

Past that everything on BOTH sides is just guesswork.

canon rumors FORUM

Re: Why so long on the DXOMark and DPReviews?
« Reply #42 on: April 13, 2012, 11:47:27 AM »

JR

  • 1D X
  • *******
  • Posts: 1244
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #43 on: April 13, 2012, 01:14:09 PM »


Past that everything on BOTH sides is just guesswork.
[/quote]

Maybe some educated guess  ;) but well said!
1DX, 24mm f1.4L II, 35mm f1.4L, 50mm f1.2L, 85mm f1.2L II, 135mm f2L, 24-70mm f2.8L II, 70-200mm f2.8L IS II :  D800, D4, and a whole bunch of Nikon lenses

well_dunno

  • 7D
  • *****
  • Posts: 356
    • View Profile
Re: Why so long on the DXOMark and DPReviews?
« Reply #44 on: April 17, 2012, 11:56:23 AM »
Mark III results will be on DxO within the next few days it seems...

http://forum.dxomark.com/index.php?topic=720.15

canon rumors FORUM

Re: Why so long on the DXOMark and DPReviews?
« Reply #44 on: April 17, 2012, 11:56:23 AM »