September 14, 2014, 10:23:57 PM

Author Topic: Can anyone explain what on Earth DXOMark is on about?  (Read 8282 times)

gary

  • Rebel T5i
  • ****
  • Posts: 116
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #30 on: April 05, 2013, 02:19:44 PM »
Having read the parameters for testing I have to say I am more than a little confused. Comparing apples and oranges is not very scientific unless of course you are testing to see that an orange tastes like an orange. As a camera doesn't really function well without a lens I have tended to ignore pure sensor tests so was intrigued to see a test of bodies and lens. I was then sadly let down when I didn't see side by side like for like comparisons. I think I will just stick to looking at the photo's I take and make my own judgement.
5D Mkiii -  16-35L mkii, 24-70L mkii, 70-200L IS mkii, 85 1.8

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #30 on: April 05, 2013, 02:19:44 PM »

Sporgon

  • 1D X
  • *******
  • Posts: 1868
  • 5% of gear used 95% of the time
    • View Profile
    • www.buildingpanoramics.com
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #31 on: April 05, 2013, 02:49:58 PM »
Quote
Since I had +1 on Sporgon's post, I feel I should mention it - I think you've read Sporgon's post in a hurry. he was not speaking of his own assumptions but was being humorous with respect to another poster who had found the DXOMark article "disturbing" and has had a rather large axe to grind on these forums lately.

Fair enough - I should have stated 'false assumptions' rather than 'your false assumptions'. Sorry about that!


That's OK. If I may quote Benedick from 'Much Ado About Nothing'

"In a false quarrel there is no true valour"  ;D

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14308
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #32 on: April 05, 2013, 06:02:43 PM »
with a good lens, and there are many, high resolution will allways be better than less.

Of course. But the point of the DxOMark wasn't 'with a good lens' it was an average across a group of lenses. 

I do find it quite interesting that for some common lenses like the 'workhorse' 24-70 f/2.8, FF 'kit' 24-105/120 f/4, and 100/105 Macro, for sharpness the Canon lens on the 5DIII outperforms the respective Nikon lens on the D800.  The first two pairs are 'general purpose zooms' and are arguably some of the most commonly used lenses, the 3rd pair are the most common macro lenses. In those cases, with these common lenses, the D800 offers no resolution advantage over the 5DIII.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

jthomson

  • EOS M2
  • ****
  • Posts: 157
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #33 on: April 05, 2013, 06:16:55 PM »
I'm curious...what is disturbing about the way the results are presented?  Are you equally 'disturbed' by their presentation of the results that show 14.4 stops of DR for the D800?  ::)

What disturbs me is the lack of apples to apples comparisons.  It would have been good if they had at least given some comparisons of the two camera's with the same lens.  If I'm reading the data correctly the Canon 5D3 with the Sigma 85 f1.4 has the same DxO score as the D800 with the same lens.  The data give the sharpness of the Canon combo in their top ten, but not the sharpness of the Nikon Combo.  Does this mean the Canon combo is sharper that the nikon combo or did they just not test the nikon combo.  All their top ten sharpness combo's tell me is that the Canon 300mm f2.8L is one great piece of glass.   

It would also have been informative if they had tested some of the nikon lenses on Canon bodies.  What happens with a nikkor 85mm f1.4 G on a 5D3. 

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #34 on: April 05, 2013, 06:49:51 PM »
DXOMark's biggest failing is not the data that they generate from their measurements, it's the absurd way that they seem to calculate their single number scores. 

While that's true of some of their Scores, I view P-Mpix as more of a measurement than a single-number score.  It would be nice if they came out and stated that's it's bascially SQF scaled to a megapixel-like value.  Based on their description and references to I3A, P-Mpix pretty clearly is an SQF-like measurement.  That's entirely consistent with a part of their business that generates a significant portion of their revenue - assessing mobile phone camera performance (one of the I3A's initiatives is CPIQ, a working group of the IEEE that's establishing a standard for Camera Phone Image Quality). 

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14308
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #35 on: April 05, 2013, 07:19:57 PM »
well , it must be some errors

As far as I know, DxO tests just one copy of a lens.  Given Sigma's reputed less-than-stellar QC (not than Canon/Nikon are perfect, far from it). So, perhaps not errors on DxO's part, but rather copy variability as seen in Roger Cicala's testing of his rental stock.

P-Mpix is actually a computed number (well, MTF is computed as well but P-Mpix is computed from the MTF data). How is it computed - it is a well kept secret. Until they explain what it is, it is worthless.

SQF is one of the many attempts to represent the whole MTF curve by one number. It depends on a reference viewing size. As such, it is not better than, say, MTF-50.

I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #36 on: April 05, 2013, 07:26:55 PM »
I wouldn't say worthless - unlike their Sensor Score, which introduces bias with the method, nothing in the description of P-Mpix suggests it's other than a linear transform of SQF measurements.

I'd say SQF is not necessarily better than MTF50 as a single-number comparator, but perhaps more relevant/useful, in that the values more closely approximate how a viewer would rate picture sharpness (which isn't always the case for MTF50).

At what distance, and what print size?

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #36 on: April 05, 2013, 07:26:55 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14308
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #37 on: April 05, 2013, 07:45:28 PM »
At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height).  Granted, there's no guarantee they're following convention...
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #38 on: April 05, 2013, 08:03:59 PM »
At what distance, and what print size?

The convention for SQF is to render an arbitrary selection of distance and print size moot by relating them with viewing distance ∝ √(picture height).  Granted, there's no guarantee they're following convention...

What does "∝" stand for? If it means "equal", then this depends on the units  (m, cm, ft) used. If means "proportional", then SQF depends on the constant. In all cases, there is some choice of distance/print size incorporated.

The way it is calculated, it is some kind of weighted average of the MTF values. Well, what is wrong with those values in the first place as a way to report what they measure? Any attempt to represent a whole function by one number is a failure.


neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14308
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #39 on: April 05, 2013, 08:22:36 PM »
What does "∝" stand for? If it means "equal", then this depends on the units  (m, cm, ft) used. If means "proportional", then SQF depends on the constant.
The ∝ symbol is mathematical shorthand for 'is directly proportional to'. The full relationship (again, by convention) for viewing distance (d) and picture height (PH)  is: d = 30cm x √(PH /10).

I do agree that representing a complex optical system with a single number is over simplifying. Have you seen Bob Atkins' illustration of that concept?  He makes the point quite well with the analogy of representing an image as its average color:

EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Don Haines

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3156
  • Posting cat pictures on the internet since 1986
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #40 on: April 05, 2013, 10:34:18 PM »
As far as I know, DxO tests just one copy of a lens.  Given Sigma's reputed less-than-stellar QC (not than Canon/Nikon are perfect, far from it). So, perhaps not errors on DxO's part, but rather copy variability as seen in Roger Cicala's testing of his rental stock.

This is my problem with DxO... to only test one sample is to get a meaningless number...... did they test the best one ever produced or the worst? What is the range? What is typical? It's a lot like looking at the road, the first car that drives by is red, and saying that all cars are red......

Roger Cicala tests dozens of copies of a lens... he can tell you what the best and worst are and where you should reasonably expect a "normal" copy to lie.
The best camera is the one in your hands

Aglet

  • 1D Mark IV
  • ******
  • Posts: 973
    • View Profile
Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #41 on: April 06, 2013, 12:24:23 AM »
So, while more is awesome, can't we be happy with what we've already got?

as long as options exist,
never, ever, settle.
Strive for better.
/philosophy

canon rumors FORUM

Re: Can anyone explain what on Earth DXOMark is on about?
« Reply #41 on: April 06, 2013, 12:24:23 AM »