July 29, 2014, 03:24:44 AM

Author Topic: Can anyone explain what on Earth DXOMark is on about?  (Read 7869 times)


canon rumors FORUM

Can anyone explain what on Earth DXOMark is on about?
« on: April 05, 2013, 07:21:22 AM »

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #1 on: April 05, 2013, 07:27:50 AM »
This has to be DXOMark's most bizzarely written article ever (and it has a lot of competition!)

http://www.dxomark.com/index.php/Publications/DxOMark-Reviews/Which-lenses-should-you-choose-for-your-Canon-EOS-5D-Mark-III/Canon-EOS-5D-Mark-III-vs.-Nikon-D800-Competition-is-closer-than-expected


Weird. It basically contradicts everything they've been publishing so far. Maybe they are a bit concerned about loosing popularity among Canon users and are stepping back?

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13594
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #2 on: April 05, 2013, 07:37:42 AM »
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Sporgon

  • 1D X
  • *******
  • Posts: 1696
  • 5% of gear used 95% of the time
    • View Profile
    • www.buildingpanoramics.com
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #3 on: April 05, 2013, 07:40:08 AM »
Maybe they have come to realise that they are leaving the real world behind.

Obviously a lot of work goes into their 'testing', but they don't do themselves any favours IMO, from lumping contradictory factors together and producing an 'average' score.

Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13594
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #4 on: April 05, 2013, 07:52:29 AM »
Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

traveller

  • 5D Mark III
  • ******
  • Posts: 653
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #5 on: April 05, 2013, 07:57:02 AM »
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

"In future tests, it will be interesting to see if the Sony sourced sensor in the Nikon D800E variant with its altered (zero strength) OLPF (Optical Low-Pass Filter) is significantly more efficient at resolving detail or if it’s as a result of the differences in fill-factor (affected by RGB filter transmission, micro-lens design and circuitry) between the Canon and Nikon sensors.

Either way, the Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well.
" [DXOMark]

I think they're saying more than that; it seems that they're implying that there is a difference in the architecure of the 5D Mark III sensor that means it resolves more detail 'per pixel' than either the D800 or previous Canon full frame sensors. 

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #6 on: April 05, 2013, 08:04:29 AM »
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

If you compare a 3rd party lens - let's take the Zeiss 100 MP, which is known to be pretty sharp - on 5D3 and D800, they rate it sharper on the 5D3 - which makes no sense. However, its total score is lower than if combined with the D800 - which also makes no sense.

However, one explanation can be:
Quote
Sharpness is a subjective quality attribute of an image or a lens. Sharpness indicates the visually perceived quality of details of an image or details reproduced by a lens. It is associated with both resolution and contrast of reproduced details (within an image or by a lens).
The DxOMark score for Sharpness is based on the Perceptual Megapixel (P-Mpix) concept that weights the Modulation Transfer Function (MTF) of the lens with the human visual acuity. Read more about Perceptual Megapixels.

If they take contrast and thus "subjective perceptions" into account, it all looses any scientific relevance. They were better off measuring sensor parameters alone.

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #6 on: April 05, 2013, 08:04:29 AM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13594
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #7 on: April 05, 2013, 08:15:58 AM »
While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

cocopop05

  • Rebel SL1
  • ***
  • Posts: 87
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #8 on: April 05, 2013, 08:26:56 AM »
It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said. 

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings. 

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #9 on: April 05, 2013, 08:38:02 AM »
While there are certainly some inconsistencies in the data from DxOMark (most of which I expect come down to copy variation among tested lenses), in general their results are accurately measured and reported.

For those who think this is some kind of about-face that contradicts their previous measurements, keep in mind that their Sensor Score (on which the D800 trounced the 5DIII) does not consider sharpness at all, whereas these recently posted results consider only sharpness.

The real about-face here is more likely to be from people who believe or disbelieve DxO on the grounds that they are biased toward Nikon sensor superiority.

I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

It's simple.  The D800 has a better sensor than the 5DIII. Canon, in aggregate and on average, has better lenses than Nikon (in terms of sharpness, at least).  The resolution we care about is the system performance = sensor + lens.  So, while the D800 has a better sensor than the 5DIII, once you slap a lens on both cameras and take pictures, the resolution differences are a wash (on average across a large set of lenses, obviously specific lenses will vary).  I and others have made statements to that effect before, DxO is just quantifying those statements.

Perfectly said. 

And DXO I suspect would not care about the opinion of Canon users, they simply test equipment and publish their findings. 

DxO is not a public service or a no-profit. They are a company, and as such they have all the interest in publishing things that will increase their popularity among users of the market leader company.


Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #10 on: April 05, 2013, 09:00:54 AM »
I think they fall victim of their own undocumented P-Mpix metric. Some of the claims are absurd, or trivial. I have seen many comparisons (for example, on IR) which show the D800 resolving more than the 5D3 without any doubt, with lenses which are not necessarily state of the art.

"Nikon D800 sensor simply isn’t as adept at resolving detail, pixel-by-pixel, as the Canon EOS 5D Mark III. There’s another surprise as well." - I m surprised that they are surprised. The 7D is not resolving as much as the 350D pixel by pixel but resolves more on image level. Why would anybody be surprised by that?

Sporgon

  • 1D X
  • *******
  • Posts: 1696
  • 5% of gear used 95% of the time
    • View Profile
    • www.buildingpanoramics.com
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #11 on: April 05, 2013, 09:01:21 AM »
Case in point - lenses. The scores are weighted in favour of faster lenses - 'transmission' - but a f1.4 lens is not necessarily better than a f2.8 one in practice. 'Transmission' is a moot point in still photography at best.

But in this case (the referenced article), your point is irrelevant because they're using their P-mpix value, their measure of sharpness.


OK, well perhaps it would be more relevant to suggest that this test may have been prompted by Roger at Lens Rental's comparison between the D800 and 5D with various standard zooms - reported here on CR.

Anyway, within sensors of the same size, would you not expect one with  larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13594
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #12 on: April 05, 2013, 09:11:14 AM »
I recall Roger Cicala's measurements finding that the Zeiss 100 MP performs better on the D800E than on the D800. I still can't find a logic explanation for a lens that's capable of outresolving a 36MP sensor to deliver better detail on a 22 MP sensor - which is also known to have a strongish AA filter.

Results will always depend on what's being measured.  Roger uses Imatest and presents MTF50 data.  That's a useful comparator benchmark in some ways, but a poor one in others.  DxOMark now uses P-Mpix, which considers acutance and contrast in addition to MTF, and better approximates human perception of sharpness.  It's worth noting that DxO's P-MPix is similar to the Subjective Quality Factor (SQF) that Imatest incorporated a few years ago (the concept of SQF has been around for 40 years), for the same reason - MTF50 doesn't tell the whole story when it comes to sharpness .
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #12 on: April 05, 2013, 09:11:14 AM »

Albi86

  • 1D Mark IV
  • ******
  • Posts: 837
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #13 on: April 05, 2013, 09:16:14 AM »

Anyway, within sensors of the same size, would you not expect one with  larger pixels to exhibit greater per pixel sharpness than one with much smaller pixels, all else being equal ?

Yes, this is why the 5D2 also scores pretty well in this regard. Similar pixel size to the 5D3 but weaker AA filter, as far as I know.

What's disturbing about this article is the way results are presented.

TrumpetPower!

  • 1D Mark IV
  • ******
  • Posts: 951
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #14 on: April 05, 2013, 09:19:17 AM »
If I read them right, they're basically claiming that all those extra megapickles of the D800 just get you larger files, but they don't get you any sharper prints.

I can believe that without trouble.

Consider all those who've compared 1.6x cropped images from the 5DIII with the 7D and found the 5DIII to be the clear winner. That's the exact same type of comparison, only with an even more dramatic difference in mepickle count.

If DXO's numbers are are all meaningful, then they should discover the 1DX absolutely trouncing the D800 on their sharpies-per-megapickle score. If they don't discover that, then this is nothing more than yet another bizarro-world imaginary number pulled from their collective netherbits.

I gotta say, though, this is another one of those bizarre "My minivan has a top speed of 110 mph but yours has a limiter that kicks in at 95 mph, so neener neener" types of things. You can already make absolutely stunning 24" x 36" prints from any of these full-frame cameras with ease, and you can max out the width of a 44" printer with a single exposure with care so long as viewers won't actually physically be sticking their noses in them. And for what those kinds of prints sell for, if you really need to deliver them with any type of regularity, you can easily afford the medium format gear that's actually designed to do that sort of thing.

I'm not at all exaggerating when I state that a 5DIII with a TS-E 24 has significantly more image quality than anything Ansel Adams ever laid hands on, including his 8" x 10" view camera.

So, while more is awesome, can't we be happy with what we've already got?

Cheers,

b&

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #14 on: April 05, 2013, 09:19:17 AM »