canon rumors FORUM

Gear Talk => EOS Bodies - For Stills => Topic started by: traveller on April 05, 2013, 07:21:22 AM

Title: Can anyone explain what on Earth DXOMark is on about?
Post by: traveller on April 05, 2013, 07:21:22 AM
This has to be DXOMark's most bizzarely written article ever (and it has a lot of competition!)

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected (http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected)
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Albi86 on April 05, 2013, 07:27:50 AM
This has to be DXOMark's most bizzarely written article ever (and it has a lot of competition!)

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected (http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected)

Weird. It basically contradicts everything they've been publishing so far. Maybe they are a bit concerned about loosing popularity among Canon users and are stepping back?
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 07:37:42 AM
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Sporgon on April 05, 2013, 07:40:08 AM
Maybe they have come to realise that they are leaving the real world behind.

Obviously a lot of work goes into their 'testing', but they don't do themselves any favours IMO, from lumping contradictory factors together and producing an 'average' score.

Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 07:52:29 AM
Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: traveller on April 05, 2013, 07:57:02 AM
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

"In future tests, it will be interesting to see if the Sony sourced sensor in the Nikon D800E variant with its altered (zero strength) OLPF (Optical Low-Pass Filter) is significantly more efficient at resolving detail or if it’s as a result of the differences in fill-factor (affected by RGB filter transmission, micro-lens design and circuitry) between the Canon and Nikon sensors.

Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well." [DXOMark]

I think they're saying more than that; it seems that they're implying that there is a difference in the architecure of the 5D Mark III sensor that means it resolves more detail 'per pixel' than either the D800 or previous Canon full frame sensors. 
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Albi86 on April 05, 2013, 08:04:29 AM
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

If you compare a 3rd party lens - let's take the Zeiss 100 MP, which is known to be pretty sharp - on 5D3 and D800, they rate it sharper on the 5D3 - which makes no sense. However, its total score is lower than if combined with the D800 - which also makes no sense.

However, one explanation can be:
Quote
Sharpness is a subjective quality attribute of an image or a lens. Sharpness indicates the visually perceived quality of details of an image or details reproduced by a lens. It is associated with both resolution and contrast of reproduced details (within an image or by a lens).
The DxOMark score for Sharpness is based on the Perceptual Megapixel (P-Mpix) concept that weights the Modulation Transfer Function (MTF) of the lens with the human visual acuity. Read more about Perceptual Megapixels.

If they take contrast and thus "subjective perceptions" into account, it all looses any scientific relevance. They were better off measuring sensor parameters alone.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 08:15:58 AM
While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: cocopop05 on April 05, 2013, 08:26:56 AM
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said. 

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings. 
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Albi86 on April 05, 2013, 08:38:02 AM
While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.

I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said. 

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings. 

DxO is not a public service or a no-profit. They are a company, and as such they have all the interest in publishing things that will increase their popularity among users of the market leader company.

Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Pi on April 05, 2013, 09:00:54 AM
I think they fall victim of their own undocumented P-Mpix metric. Some of the claims are absurd, or trivial. I have seen many comparisons (for example, on IR) which show the D800 resolving more than the 5D3 without any doubt, with lenses which are not necessarily state of the art.

"Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well." - I m surprised that they are surprised. The 7D is not resolving as much as the 350D pixel by pixel but resolves more on image level. Why would anybody be surprised by that?
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Sporgon on April 05, 2013, 09:01:21 AM
Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.


OK, well perhaps it would be more relevant to suggest that this test may have been prompted by Roger at Lens Rental's comparison between the D800 and 5D with various standard zooms - reported here on CR.

Anyway, within sensors of the same size, would you not expect one with  larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 09:11:14 AM
I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

Results will always depend on what's being measured.  Roger uses Imatest and presents MTF50 data.  That's a useful comparator benchmark in some ways, but a poor one in others.  DxOMark now uses P-Mpix, which considers acutance and contrast in addition to MTF, and better approximates human perception of sharpness.  It's worth noting that DxO's P-MPix is similar to the Subjective Quality Factor (SQF) that Imatest incorporated a few years ago (the concept of SQF has been around for 40 years), for the same reason - MTF50 doesn't tell the whole story when it comes to sharpness .
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Albi86 on April 05, 2013, 09:16:14 AM

Anyway, within sensors of the same size, would you not expect one with  larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?

Yes, this is why the 5D2 also scores pretty well in this regard. Similar pixel size to the 5D3 but weaker AA filter, as far as I know.

What's disturbing about this article is the way results are presented.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: TrumpetPower! on April 05, 2013, 09:19:17 AM
If I read them right, they're basically claiming that all those extra megapickles of the D800 just get you larger files, but they don't get you any sharper prints.

I can believe that without trouble.

Consider all those who've compared 1.6x cropped images from the 5DIII with the 7D and found the 5DIII to be the clear winner. That's the exact same type of comparison, only with an even more dramatic difference in mepickle count.

If DXO's numbers are are all meaningful, then they should discover the 1DX absolutely trouncing the D800 on their sharpies-per-megapickle score. If they don't discover that, then this is nothing more than yet another bizarro-world imaginary number pulled from their collective netherbits.

I gotta say, though, this is another one of those bizarre "My minivan has a top speed of 110 mph but yours has a limiter that kicks in at 95 mph, so neener neener" types of things. You can already make absolutely stunning 24" x 36" prints from any of these full-frame cameras with ease, and you can max out the width of a 44" printer with a single exposure with care so long as viewers won't actually physically be sticking their noses in them. And for what those kinds of prints sell for, if you really need to deliver them with any type of regularity, you can easily afford the medium format gear that's actually designed to do that sort of thing.

I'm not at all exaggerating when I state that a 5DIII with a TS-E 24 has significantly more image quality than anything Ansel Adams ever laid hands on, including his 8" x 10" view camera.

So, while more is awesome, can't we be happy with what we've already got?

Cheers,

b&
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Rienzphotoz on April 05, 2013, 09:22:33 AM
Interesting :-\
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: traveller on April 05, 2013, 09:25:59 AM
I think they fall victim of their own undocumented P-Mpix metric....

I think you're probably right; GIGO, as they say!  ;)

DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 09:34:46 AM
DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score.  It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value.  Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement.  That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality). 

What's disturbing about this article is the way results are presented.

I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Sporgon on April 05, 2013, 09:53:30 AM

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: TrumpetPower! on April 05, 2013, 10:15:42 AM

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Or, in other words, if it was a cheese shop, it wouldn't be much of one, would it, now -- even if it was the finest one in the lineup. So clean, really -- in the sense, of course, that it's uncontaminated by photos.

b&
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: J.R. on April 05, 2013, 10:20:38 AM

Quote
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

+1  ;D ;D ;D
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 10:32:47 AM
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: traveller on April 05, 2013, 10:49:05 AM
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores. [DXOMark]

Is DXOMark confusing the issue, or are they just confusing me!  ???



Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: J.R. on April 05, 2013, 10:53:33 AM
I would think that what he finds disturbing is the fact that DXO rate the D800 sensor much higher than the 5D, but then say;

" 'half a mo, when we put a lens on it we're not so sure"....... ::) ::)

Show me where their sensor measurements include sharpness as a tested parameter.  Show me where the above article incorporates the measures they do use for sensors (DR, color depth, ISO noise) in the P-Mpix measurement used for the plot?

It's like they reported data on two people, one being 180 cm tall and the other being 160 cm tall, and you find it disturbing when they later report that the 160 cm person weighs 90 kg while the 180 cm person weighs 75 kg.  You assumed the taller person would weigh more, and you're 'disturbed' when the data don't fit your assumptions.  The problem is your false assumptions, not the reported data.

Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: mitchell3417 on April 05, 2013, 11:37:11 AM
All this does it put numbers to what we already knew. When you put a lens on a D800 and a 5DIII and go out and take pictures the difference in IQ (specifically resolution in this case) is not as large as some people would have you believe. Both cameras are great. It's not worth arguing over. The only reason one needs to compare cameras is if one is going to invest in a new camera system. In that case, one should rent the cameras in question and decide for himself which one best fits his needs.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 11:55:44 AM
Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!

So what do they mean when they write this, with their Sensor Scores graph below?

Even the DxOMark mean and median scores can be explained by the small difference (-1/3 stop) in the Low Light ISO score between the cameras.

A small (1/3 stop) difference in the Low Light ISO score between the cameras accounts for lower median and mean DxO Mark camera / lens scores. [DXOMark]

Is DXOMark confusing the issue, or are they just confusing me!  ???

I think it's them, you have a point about the article itself.  They start off and conclude with what I think is the central point of the article, illustrated by these data (as a side note, I'd love it if this plot was 'live' i.e. you could click a data point and see the lens it represents):

(http://cdn.dxomark.com/var/ezwebin_site/storage/images/media/images/graph13/80780-2-eng-US/graph1.jpg)

But, after presenting that central piece of information, they felt compelled to discuss why that differs somewhat if instead of just looking at sharpness (P-Mpix), you look at their summary 'DxOMark Lens Scores' (and you may recall how I feel about their Overall Sensor Score...the Lens Score not really any better).  But still, they could have simply stated, "The above comparison is in regards to sharpness on the 5DIII vs. D800, and the Lens Score considers parameters other than sharpness, including low light performance on a given body," and been done with that bit.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Mt Spokane Photography on April 05, 2013, 12:02:20 PM
One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.
 
They have started testing lenses a while back, and publishing results in conjunction with DPR.  I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.
 
We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%.  This is due to a lens resolution, circle of confusion, and other factors.  When you get high pixel densities, resolution of the system does not scale, but is always a improvement.
 
That is what DXO mark is saying.
 
Technically,  the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link.  Even film has MTF values specified.
 
So, for simplicity,  if a lens has a MTF of 0.9, a Body 0.8,  together they are 0.72.  Increase the body to 0.85, and the system becomes 0.765.  Better, but not a revolution because the lens needs to get better as well.
 
However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).
 
One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.
 
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Aglet on April 05, 2013, 12:16:31 PM
One of my complaints about DXO has been that they measure a sensor and declare a camera to be the best without ever taking a photo.
 
They have started testing lenses a while back, and publishing results in conjunction with DPR.  I suspect that this has raised questions as to what the rating of a camera would be if a lens were installed.
 
We have long noted that a image from 6 or 10 mp sensor looks very sharp when viewed at 100%.  This is due to a lens resolution, circle of confusion, and other factors.  When you get high pixel densities, resolution of the system does not scale, but is always a improvement.
 
That is what DXO mark is saying.
 
Technically,  the MTF of the system is equal to the MTF of the individual components multiplied together, and is always less than the weakest link.  Even film has MTF values specified.
 
So, for simplicity,  if a lens has a MTF of 0.9, a Body 0.8,  together they are 0.72.  Increase the body to 0.85, and the system becomes 0.765.  Better, but not a revolution because the lens needs to get better as well.
 
However, there are those who only look at one part of the picture, the number of MP, for example and happily believe that with twice the MP, they get twice the resolution. (No reflection on CR members who mostly know better).
 
One thing that the D800 sensor brings is very noticeable improvement in dynamic range under harsh lighting conditions such as bright sun and deep shadows.

That's about right

Or, what DxOmark's testing translates to in this regard is simply

DIMINISHING RETURNS of increasing sensor resolution

Adding 60% more MP does not give correspondingly (square root of ratio) more resolution.

Adding more MP can increase overall acuity, since it's part of an equation containing the factors mentioned above, but it's only one factor.

another good example is in DPreview's comparison of the AA-less D7100 and the AA-equipped D5200, both with very similar sensors of equivalent pixel count.

www.dpreview.com/previews/nikon-d7100/6 (http://www.dpreview.com/previews/nikon-d7100/6)

only in the small range of optimal lens performance (wider apertures) can a difference be noticed whereas regular shooting with real-world lenses and settings this AA-less sensor performance is no better.
(personally I'd take the d5200 over the 7100 for better performance/cost, slightly lower overall sensor noise and it oddly fit my hand better than the 7100)
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Albi86 on April 05, 2013, 12:36:12 PM

I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

I'm not disturbed by that because I'm aware it's an approximation deriving from their test methodology. In fact I'm not very interested in the absolute value, but instead I find it relevant in relative terms - I.e. camera x vs camera y.

What I find disturbing is the vaguely (yet apparent, at least to me) apologetic tone of this article. Examples:

Quote
5D Mark III has a highly sensitive and accurate 61-point focusing system, durable pro-level build, excellent live view implementation, and can capture stills at up to 6fps (compared with 4.5 fps for the Nikon).

When did they start taking such things into account?

Quote
When using specific lenses (such as the new Sigma 35mm f/1.4 DG HSM A) the Nikon can out resolve the Canon. However, taken as a whole, the statistics reveal the EOS 5D MK III is capable of similar sharpness and of achieving a close DxOMark camera/lens score to the Nikon D800. Moreover, that’s despite the latter camera’s 60% extra pixel count.

So... Are we comparing cameras, or camera systems, or what? Because saying that a 5D3 with a good lens is better than a D800 with a crappy lens doesn't sound like a great revelation to me.

Quote
While that same principle of choosing the best glass also applies to the Canon, it’s not quite as crucial. In financial terms alone, this may be an important consideration if you have already invested in a sizeable range of lenses. Lenses on the Canon EOS 5D Mark III are without doubt very efficient.

Why not keeping the old D700 then? Even less demanding with only 12 mp, and it also makes you save some grands on a new camera!

Quote
Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well.

Here a comparison with the D600/D700 would have been nice.

Quote
If you’re a Canon user and are looking to upgrade, or maybe even switch because of the sensor’s perceived lowly capabilities then this should put your mind at rest.

That's where they wanted to get. All the rest IMHO is a filler.

So yes, I preferred when they just tested sensor performance. That's the only thing that can be done in a scientific and easily comparable way. Adopting this sterile approach on more subjective matters is just misleading IMHO. And this article feels very bizarre to me in its style, content and conclusions.




Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: David Hull on April 05, 2013, 02:12:43 PM
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Yep... I think it really just comes down to that.  People need to think in terms of the system and not just the individual components.  That is probably the only take away from all of this.

I found it interesting that with the so-called "Holy Trinity" (which is what most pros would be packing or at least lusting after) i.e. the 16-35L II, the 24-70L II and the 70-200LII,  produces better performance when combined with the 5DIII than the Nikon equivalents do when combined with the D800.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: gary on April 05, 2013, 02:19:44 PM
Having read the parameters for testing I have to say I am more than a little confused. Comparing apples and oranges is not very scientific unless of course you are testing to see that an orange tastes like an orange. As a camera doesn't really function well without a lens I have tended to ignore pure sensor tests so was intrigued to see a test of bodies and lens. I was then sadly let down when I didn't see side by side like for like comparisons. I think I will just stick to looking at the photo's I take and make my own judgement.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Sporgon on April 05, 2013, 02:49:58 PM
Quote
Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!


That's OK. If I may quote Benedick from 'Much Ado About Nothing'

"In a false quarrel there is no true valour"  ;D
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 06:02:43 PM
with a good lens, and there are many, high resolution will allways be better than less.

Of course. But the point of the DxOMark wasn't 'with a good lens' it was an average across a group of lenses. 

I do find it quite interesting that for some common lenses like the 'workhorse' 24-70 f/2.8, FF 'kit' 24-105/120 f/4, and 100/105 Macro, for sharpness the Canon lens on the 5DIII outperforms the respective Nikon lens on the D800.  The first two pairs are 'general purpose zooms' and are arguably some of the most commonly used lenses, the 3rd pair are the most common macro lenses. In those cases, with these common lenses, the D800 offers no resolution advantage over the 5DIII.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: jthomson on April 05, 2013, 06:16:55 PM
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

What disturbs me is the lack of apples to apples comparisons.  It would have been good if they had at least given some comparisons of the two camera's with the same lens.  If I'm reading the data correctly the Canon 5D3 with the Sigma 85 f1.4 has the same DxO score as the D800 with the same lens.  The data give the sharpness of the Canon combo in their top ten, but not the sharpness of the Nikon Combo.  Does this mean the Canon combo is sharper that the nikon combo or did they just not test the nikon combo.  All their top ten sharpness combo's tell me is that the Canon 300mm f2.8L is one great piece of glass.   

It would also have been informative if they had tested some of the nikon lenses on Canon bodies.  What happens with a nikkor 85mm f1.4 G on a 5D3. 
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Pi on April 05, 2013, 06:49:51 PM
DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score.  It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value.  Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement.  That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality). 

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 07:19:57 PM
well , it must be some errors

As far as I know, DxO tests just one copy of a lens.  Given Sigma's reputed less-than-stellar QC (not than Canon/Nikon are perfect, far from it). So, perhaps not errors on DxO's part, but rather copy variability as seen in Roger Cicala's testing of his rental stock.

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.

I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Pi on April 05, 2013, 07:26:55 PM
I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).

At what distance, and what print size?
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 07:45:28 PM
At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height).  Granted, there's no guarantee they're following convention...
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Pi on April 05, 2013, 08:03:59 PM
At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height).  Granted, there's no guarantee they're following convention...

What does "∝" stand for? If it means "equal", then this depends on the units  (m, cm, ft) used. If means "proportional", then SQF depends on the constant. In all cases, there is some choice of distance/print size incorporated.

The way it is calculated, it is some kind of weighted average of the MTF values. Well, what is wrong with those values in the first place as a way to report what they measure? Any attempt to represent a whole function by one number is a failure.

Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: neuroanatomist on April 05, 2013, 08:22:36 PM
What does "∝" stand for? If it means "equal", then this depends on the units  (m, cm, ft) used. If means "proportional", then SQF depends on the constant.
The ∝ symbol is mathematical shorthand for 'is directly proportional to'. The full relationship (again, by convention) for viewing distance (d) and picture height (PH)  is: d = 30cm x √(PH /10).

I do agree that representing a complex optical system with a single number is over simplifying. Have you seen Bob Atkins' illustration of that concept?  He makes the point quite well with the analogy of representing an image as its average color:

(http://www.bobatkins.com/photography/technical/mtf/mona2.jpg)
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Don Haines on April 05, 2013, 10:34:18 PM
As far as I know, DxO tests just one copy of a lens.  Given Sigma's reputed less-than-stellar QC (not than Canon/Nikon are perfect, far from it). So, perhaps not errors on DxO's part, but rather copy variability as seen in Roger Cicala's testing of his rental stock.

This is my problem with DxO... to only test one sample is to get a meaningless number...... did they test the best one ever produced or the worst? What is the range? What is typical? It's a lot like looking at the road, the first car that drives by is red, and saying that all cars are red......

Roger Cicala tests dozens of copies of a lens... he can tell you what the best and worst are and where you should reasonably expect a "normal" copy to lie.
Title: Re: Can anyone explain what on Earth DXOMark is on about?
Post by: Aglet on April 06, 2013, 12:24:23 AM
So, while more is awesome, can't we be happy with what we've already got?

as long as options exist,
never, ever, settle.
Strive for better.
/philosophy