September 21, 2014, 12:30:17 PM

Author Topic: Can anyone explain what on Earth DXOMark is on about?  (Read 8335 times)

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3322
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #15 on: April 05, 2013, 09:22:33 AM »
Interesting :-\
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #15 on: April 05, 2013, 09:22:33 AM »

traveller

  • 5D Mark III
  • ******
  • Posts: 657
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #16 on: April 05, 2013, 09:25:59 AM »
I think they fall victim of their own undocumented P-Mpix metric....

I think you're probably right; GIGO, as they say!  ;)

DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14446
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #17 on: April 05, 2013, 09:34:46 AM »
DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score.  It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value.  Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement.  That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality). 

What's disturbing about this article is the way results are presented.

I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Sporgon

  • 1D X
  • *******
  • Posts: 1886
  • 5% of gear used 95% of the time
    • View Profile
    • www.buildingpanoramics.com
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #18 on: April 05, 2013, 09:53:30 AM »

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

TrumpetPower!

  • 1D Mark IV
  • ******
  • Posts: 951
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #19 on: April 05, 2013, 10:15:42 AM »

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Or, in other words, if it was a cheese shop, it wouldn't be much of one, would it, now -- even if it was the finest one in the lineup. So clean, really -- in the sense, of course, that it's uncontaminated by photos.

b&

J.R.

  • 1D X
  • *******
  • Posts: 1479
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #20 on: April 05, 2013, 10:20:38 AM »

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

+1  ;D ;D ;D
5D3, 6D
16-35L, 24-70L II, 70-200L II, 100-400L, 50L, 85L II, 135L, 24TSE, 40, 100 macro, 600RT x 4
I have more photo gear than I need. The blame lies squarely with Canonrumors

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14446
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #21 on: April 05, 2013, 10:32:47 AM »
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #21 on: April 05, 2013, 10:32:47 AM »

traveller

  • 5D Mark III
  • ******
  • Posts: 657
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #22 on: April 05, 2013, 10:49:05 AM »
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores.
[DXOMark]

Is DXOMark confusing the issue, or are they just confusing me!  ???




J.R.

  • 1D X
  • *******
  • Posts: 1479
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #23 on: April 05, 2013, 10:53:33 AM »
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.

Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.
5D3, 6D
16-35L, 24-70L II, 70-200L II, 100-400L, 50L, 85L II, 135L, 24TSE, 40, 100 macro, 600RT x 4
I have more photo gear than I need. The blame lies squarely with Canonrumors

mitchell3417

  • PowerShot G1 X II
  • ***
  • Posts: 69
    • View Profile
    • Mitchell Bennnett Photography
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #24 on: April 05, 2013, 11:37:11 AM »
All this does it put numbers to what we already knew. When you put a lens on a D800 and a 5DIII and go out and take pictures the difference in IQ (specifically resolution in this case) is not as large as some people would have you believe. Both cameras are great. It's not worth arguing over. The only reason one needs to compare cameras is if one is going to invest in a new camera system. In that case, one should rent the cameras in question and decide for himself which one best fits his needs.
I enjoy taking pictures!!!
flickr  MB Photo

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14446
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #25 on: April 05, 2013, 11:55:44 AM »
Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores.
[DXOMark]

Is DXOMark confusing the issue, or are they just confusing me!  ???

I think it's them, you have a point about the article itself.  They start off and conclude with what I think is the central point of the article, illustrated by these data (as a side note, I'd love it if this plot was 'live' i.e. you could click a data point and see the lens it represents):



But, after presenting that central piece of information, they felt compelled to discuss why that differs somewhat if instead of just looking at sharpness (P-Mpix), you look at their summary 'DxOMark Lens Scores' (and you may recall how I feel about their Overall Sensor Score...the Lens Score not really any better).  But still, they could have simply stated, "The above comparison is in regards to sharpness on the 5DIII vs. D800, and the Lens Score considers parameters other than sharpness, including low light performance on a given body," and been done with that bit.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8687
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #26 on: April 05, 2013, 12:02:20 PM »
One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.
 
They have started testing lenses a while back, and publishing results in conjunction with DPR.  I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.
 
We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%.  This is due to a lens resolution, circle of confusion, and other factors.  When you get high pixel densities, resolution of the system does not scale, but is always a improvement.
 
That is what DXO mark is saying.
 
Technically,  the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link.  Even film has MTF values specified.
 
So, for simplicity,  if a lens has a MTF of 0.9, a Body 0.8,  together they are 0.72.  Increase the body to 0.85, and the system becomes 0.765.  Better, but not a revolution because the lens needs to get better as well.
 
However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).
 
One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.
 

Aglet

  • 1D Mark IV
  • ******
  • Posts: 987
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #27 on: April 05, 2013, 12:16:31 PM »
One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.
 
They have started testing lenses a while back, and publishing results in conjunction with DPR.  I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.
 
We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%.  This is due to a lens resolution, circle of confusion, and other factors.  When you get high pixel densities, resolution of the system does not scale, but is always a improvement.
 
That is what DXO mark is saying.
 
Technically,  the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link.  Even film has MTF values specified.
 
So, for simplicity,  if a lens has a MTF of 0.9, a Body 0.8,  together they are 0.72.  Increase the body to 0.85, and the system becomes 0.765.  Better, but not a revolution because the lens needs to get better as well.
 
However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).
 
One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.

That's about right

Or, what DxOmark's testing translates to in this regard is simply

DIMINISHING RETURNS of increasing sensor resolution

Adding 60% more MP does not give correspondingly (square root of ratio) more resolution.

Adding more MP can increase overall acuity, since it's part of an equation containing the factors mentioned above, but it's only one factor.

another good example is in DPreview's comparison of the AA-less D7100 and the AA-equipped D5200, both with very similar sensors of equivalent pixel count.

www.dpreview.com/previews/nikon-d7100/6

only in the small range of optimal lens performance (wider apertures) can a difference be noticed whereas regular shooting with real-world lenses and settings this AA-less sensor performance is no better.
(personally I'd take the d5200 over the 7100 for better performance/cost, slightly lower overall sensor noise and it oddly fit my hand better than the 7100)

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #27 on: April 05, 2013, 12:16:31 PM »

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #28 on: April 05, 2013, 12:36:12 PM »

I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I'm not disturbed by that because I'm aware it's an approximation deriving from their test methodology. In fact I'm not very interested in the absolute value, but instead I find it relevant in relative terms - I.e. camera x vs camera y.

What I find disturbing is the vaguely (yet apparent, at least to me) apologetic tone of this article. Examples:

Quote
5D Mark III has a highly sensitive and accurate 61-point focusing system, durable pro-level build, excellent live view implementation, and can capture stills at up to 6fps (compared with 4.5 fps for the Nikon).

When did they start taking such things into account?

Quote
When using specific lenses (such as the new Sigma 35mm f/1.4 DG HSM A) the Nikon can out resolve the Canon. However, taken as a whole, the statistics reveal the EOS 5D MK III is capable of similar sharpness and of achieving a close DxOMark camera/lens score to the Nikon D800. Moreover, that’s despite the latter camera’s 60% extra pixel count.

So... Are we comparing cameras, or camera systems, or what? Because saying that a 5D3 with a good lens is better than a D800 with a crappy lens doesn't sound like a great revelation to me.

Quote
While that same principle of choosing the best glass also applies to the Canon, it’s not quite as crucial. In financial terms alone, this may be an important consideration if you have already invested in a sizeable range of lenses. Lenses on the Canon EOS 5D Mark III are without doubt very efficient.

Why not keeping the old D700 then? Even less demanding with only 12 mp, and it also makes you save some grands on a new camera!

Quote
Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well.

Here a comparison with the D600/D700 would have been nice.

Quote
If you’re a Canon user and are looking to upgrade, or maybe even switch because of the sensor’s perceived lowly capabilities then this should put your mind at rest.

That's where they wanted to get. All the rest IMHO is a filler.

So yes, I preferred when they just tested sensor performance. That's the only thing that can be done in a scientific and easily comparable way. Adopting this sterile approach on more subjective matters is just misleading IMHO. And this article feels very bizarre to me in its style, content and conclusions.





David Hull

  • EOS M2
  • ****
  • Posts: 208
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #29 on: April 05, 2013, 02:12:43 PM »
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Yep... I think it really just comes down to that.  People need to think in terms of the system and not just the individual components.  That is probably the only take away from all of this.

I found it interesting that with the so-called "Holy Trinity" (which is what most pros would be packing or at least lusting after) i.e. the 16-35L II, the 24-70L II and the 70-200LII,  produces better performance when combined with the 5DIII than the Nikon equivalents do when combined with the D800.

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #29 on: April 05, 2013, 02:12:43 PM »