October 21, 2014, 03:51:38 AM

Author Topic: Why I don't like DxO Mark Scores  (Read 5285 times)

Bolt

  • SX60 HS
  • **
  • Posts: 8
    • View Profile
Why I don't like DxO Mark Scores
« on: April 19, 2013, 04:44:16 AM »
So, I was having a discussion with a friend of mine who is also a photographer about the accuracy of DxO Mark.

In general, he was in favour of the immediate report and scoring that DxOMark.com refers to when referring to the 'Camera Sensor Ratings' and the scores they show in their ranking list. I disagreed, claiming that the results shown on the immediate report are, in my opinion, inaccurate.

For reasons not mentioned, we were comparing the Nikon D800 and the Canon 1D X. Specifically, the Dynamic Range.

I claimed that the results that the D800 had a Dynamic Range of 14.4 Evs compared to the 1D X's 11.8 is in part true, but an inaccurate representation of the tests overall results. Here is why:


The reported Dynamic Range of the D800 is 14.4 EV. While this is true, it is a representation of the MAXIMUM possible Dynamic Range at only one ISO, not the actual performance throughout the ISO range.

The Dynamic Range of the D800 drops off at an average rate of .91 Evs every increase in ISO.
Compared to the Canon 1D X, which averages an average .46 Evs loss every increase in ISO.
To put it relatively, .46 Evs is less than the smallest loss the D800 sees in an increase in ISO. 

After three increases in ISO, the D800 has lost 2.31 Evs.
Compared to the 1D X that has lost 0.16 Evs.

After five increases in ISO, the 1D X has better Dynamic Range throughout the rest of the ISO.

After nine increases in ISO [limit of D800] the D800 has dropped to half its original Dynamic Range.
Compared to the 1D X that has lost as much as the D800 did after five increases in ISO.

The average Dynamic Range throughout the ISO differs by .33 Evs
1D X - 10.60
D800 - 10.93

The deviation between highest and lowest possible levels of Dynamic Range throughout ISO differs by 90%.
1D X - 3.73
D800 - 7.33


In conclusion: [referring to Dynamic Range throughout ISO]
The results displayed on DxOMark's 'Camera Sensor Ratings' scorecards are in my opinion mis-informative of the actual results shown throughout the camera's performance. While in 'perfect' conditions the Dynamic Range of the D800 WILL outperform the 1D X, the average results, standard deviation and high-to-low end loss of performance would suggest that the 1D X is a better overall performer in regards to Dynamic Range throughout the ISO spectrum.

[These results are based only upon those given by DxOMark. I do not believe DxO is an inaccurate or dishonest company. I posted this only to represent a more in depth look at the results given on the immediate Camera Sensor Ratings scorecard. In my honest opinion, this is why I PERSONALLY believe the results shown on the Scorecard are an inaccurate representation of the camera's overall performance]

Now I can wait for the D800 lovers to tell me how much of a moron I am.

Here is a 'graph' of the results:

1DX

11.77      
11.81 +.04
11.74 -.07
11.61 -.13
11.18 -.43
10.37 -.81
9.88 -.49
8.99 -.89
8.08   -.91
   
Deviation 3.73
Average 10.60

-

D800      

14.33            
13.65 -.68         
12.83 -.82         
12.02 -.81         
11.12 -.90         
10.09 -1.03      
9.17 -.92         
8.16 -1.01         
7.00 -1.16      
   
Deviation 7.33
Average 10.93
Current: Canon 1D X, EF 24-70L F4 IS + Accessories

canon rumors FORUM

Why I don't like DxO Mark Scores
« on: April 19, 2013, 04:44:16 AM »

meli

  • EOS M2
  • ****
  • Posts: 164
    • View Profile
Re: Why I don't like DxO Mark Scores
« Reply #1 on: April 19, 2013, 07:35:10 AM »
Its one way to spin it, but your representation is actually skewed. Just by looking at the DR diagram you can see that over 3200 the 2 curvatures are almost parallel (with the 1dx clearly better) but below that 1dx's curvature is reminiscent of previous generation sensors whereas newer toshiba or sony sensors are almost a straight line (quasi isoless). 

Basically what it shows is that sony's sensor is exemplary for its category whereas 1dx's could have been better <3200


neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14709
    • View Profile
Re: Why I don't like DxO Mark Scores
« Reply #2 on: April 19, 2013, 08:30:31 AM »
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3323
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Why I don't like DxO Mark Scores
« Reply #3 on: April 26, 2013, 03:37:43 PM »
So, I was having a discussion with a friend of mine who is also a photographer about the accuracy of DxO Mark.

In general, he was in favour of the immediate report and scoring that DxOMark.com refers to when referring to the 'Camera Sensor Ratings' and the scores they show in their ranking list. I disagreed, claiming that the results shown on the immediate report are, in my opinion, inaccurate.

For reasons not mentioned, we were comparing the Nikon D800 and the Canon 1D X. Specifically, the Dynamic Range.

I claimed that the results that the D800 had a Dynamic Range of 14.4 Evs compared to the 1D X's 11.8 is in part true, but an inaccurate representation of the tests overall results. Here is why:


The reported Dynamic Range of the D800 is 14.4 EV. While this is true, it is a representation of the MAXIMUM possible Dynamic Range at only one ISO, not the actual performance throughout the ISO range.

The Dynamic Range of the D800 drops off at an average rate of .91 Evs every increase in ISO.
Compared to the Canon 1D X, which averages an average .46 Evs loss every increase in ISO.
To put it relatively, .46 Evs is less than the smallest loss the D800 sees in an increase in ISO. 

After three increases in ISO, the D800 has lost 2.31 Evs.
Compared to the 1D X that has lost 0.16 Evs.

After five increases in ISO, the 1D X has better Dynamic Range throughout the rest of the ISO.

After nine increases in ISO [limit of D800] the D800 has dropped to half its original Dynamic Range.
Compared to the 1D X that has lost as much as the D800 did after five increases in ISO.

The average Dynamic Range throughout the ISO differs by .33 Evs
1D X - 10.60
D800 - 10.93

The deviation between highest and lowest possible levels of Dynamic Range throughout ISO differs by 90%.
1D X - 3.73
D800 - 7.33


In conclusion: [referring to Dynamic Range throughout ISO]
The results displayed on DxOMark's 'Camera Sensor Ratings' scorecards are in my opinion mis-informative of the actual results shown throughout the camera's performance. While in 'perfect' conditions the Dynamic Range of the D800 WILL outperform the 1D X, the average results, standard deviation and high-to-low end loss of performance would suggest that the 1D X is a better overall performer in regards to Dynamic Range throughout the ISO spectrum.

[These results are based only upon those given by DxOMark. I do not believe DxO is an inaccurate or dishonest company. I posted this only to represent a more in depth look at the results given on the immediate Camera Sensor Ratings scorecard. In my honest opinion, this is why I PERSONALLY believe the results shown on the Scorecard are an inaccurate representation of the camera's overall performance]

Now I can wait for the D800 lovers to tell me how much of a moron I am.

Here is a 'graph' of the results:

1DX

11.77      
11.81 +.04
11.74 -.07
11.61 -.13
11.18 -.43
10.37 -.81
9.88 -.49
8.99 -.89
8.08   -.91
   
Deviation 3.73
Average 10.60

-

D800      

14.33            
13.65 -.68         
12.83 -.82         
12.02 -.81         
11.12 -.90         
10.09 -1.03      
9.17 -.92         
8.16 -1.01         
7.00 -1.16      
   
Deviation 7.33
Average 10.93
Interesting
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3323
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Why I don't like DxO Mark Scores
« Reply #4 on: April 26, 2013, 03:38:46 PM »

BTW which brand of worms are those? Canon? or Nikon?  ;D ;D ;D
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

canon rumors FORUM

Re: Why I don't like DxO Mark Scores
« Reply #4 on: April 26, 2013, 03:38:46 PM »