DxOMark vs. Reality

[Hillsilly]

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

And I think that's where the problem lies.  Very few people care or want to know about the technical side.  They just want to take great photos.  To do this you need a camera and if you're doing some research you will come across the DxO site.  How awesome is this!!  Not only do they rank all of the cameras but you can also do a comparison between them.  Despite disclaimers and fine print, you really get the impression that the DxO Mark is an objective assessment of one cameras overall ability vs another.

Anyways, I'm only anti-DxO because my camera gets a lowly "73".  Whereas my Nikon D5100 wielding sister gets an "80".  Yet my camera is sooooo much better....

[aj1575]

Actually, there is only one mistake that DxOMark makes. They should drop the sensor ranking / rating. They do a great job with all the measurements and analysis, and the graphs are very informative. But the rating does not make sense; even the new one, where they try to differentiate between portrait, landscape and high ISO does not really make sense.
Just leave the graphs there, and people can find their own conclusion.

[elflord]

Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs do not perform well at high ISOs. A "general" rating does not work with a special purpose tool.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera. 

I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same". The 7D is substantially better than the 10D for example (9 points in the overall score,  dynamic range substantiially better)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

[Canon-F1]

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.   

[straub]

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

[Maui5150]

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices 

This +1000

Most people, especially Professionals in a field, are not fools.

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

In fact, if DxOMarks is to be trusted, then their should be screams that PhaseOne are TOTAL PIECES OF CRAP and that NIKON is ripping off people by selling any camera above $3000, because the D800 is the ONLY CAMERA anyone should buy willing to spend more than $3000 and those under should go for the D600. 

Not saying they are not good cameras, but what the last year plus has highlighted to me, DxOMarks Scores are COMPLETELY IRRELEVANT AND MEANINGLESS.

An erroneous equation provides erroneous results

[straub]

what are DXO´s some would ask.   

Well, they have the word "Science" in their logo and Reichmann doesn't :-)

[Maui5150]

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 

Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations.  So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are.  Anyone can produce DATA.  But data is not information. 

What DxOMarks lacks is RELEVANCE

[M.ST]

DxOMarks are useless and only made to entertain hobby photographers.

[ankorwatt]

There are lots of posts about DxOMark, lots of support, lots of bashing, some of it fact-based, much of it hearsay and knee-jerk reaction.  There are many posts about sales figures, generally fewer than about DxOMark, and generally with even less of them with real facts.

So...let's do a little reality check, about the past 5 years in DxOMark-land and the past 5 years in the real world.

The year is 2007.  Nikon has the D3 as a flagship and the D300 as the prosumer APS-C body.  Canon has the 1DsIII and D40, respectively.  DxOMark scores the D3 as 81, the 1DsIII as 80 - a meaningless difference, by their standards.  The D300 scores a 67, while the 40D scores a 64 - that is starting to look like a real difference. 

2008 saw Nikon enter the 'affordable' FF market with the D700, which DxOMark scores an insignificant 1 point higher than Canon's 5DII at 80 vs. 79.  2009 had updates to the high-end prosumer APS-C segment, and Nikon's D300S scores a 70, while Canon's 7D scores a 66.  2010 brought us mid-range APS-C updates, and while the 60D with the 7D's sensor scores a 66, the D7000 scores an 80, beating Canon's then-current FF sensors.

This year, the somewhat-less-affordable FF battle continues, with the 5DIII scoring 81, barely better than it's predecessor, while the D800/E scores a very impressive 95/96, soundly beating all Canon cameras as well as every other sensor DxOMark has ever tested, including medium format bodies with sensors with 2.5-times larger sensors.

So, it seems quite clear that over the past 5 years in DxOMark-land, Nikon has been dramatically improving, Canon has been stagnating, and Nikon is way ahead.  But that's DxOMark-land.  What about the real world?

Canon and Nikon are publicly traded companies, and as such, they publish their annual and quarterly reports (in the Investor Relations sections of their corporate websites).  IDC (a market analysis firm) also publishes annual summaries of the market as a whole.

In 2007, Canon had 43% of the dSLR market, Nikon had 40%.  In 2010, Canon had 44.5% of the dSLR market, Nikon had 29.8%.  So, over years while Nikon was bringing us better sensors than Canon, Nikon was losing market share while Canon was gaining it.  Ok, fine, but that's 2010.  What about this year?

In 2Q2012 (by calendar year, they report it as 1QFY2013), Nikon had an 18% y/y growth of unit sales of dSLRs and lenses.  In 2Q2012, (they report by CY, not FY), Canon had a 47% y/y growth of unit sales of dSLRs and lenses.

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

What has Canon sales to do with DXO ?

DXO reports all sensor parameters that are important. FWC, QE, read noise, Color Depth and DR.
http://www.dxomark.com/index.php/Cameras/Compare-Camera-Sensors

http://home.comcast.net/~nikond70/Charts/PDR.htm  here you can look at your camera and DR

http://www.sensorgen.info     here a list over cameras and with facts like FWC, QE, read noise and dynamic range


The differences can also be seen visually by the greater exposure latitude, I do not understand why you persist in denying these facts

Regarding  MF cameras , they have more surface area but their CCD technology is light years behind the modern CMOS technology, read FWC, QE and read noise  and you will understand why

[davidpeter]

Ok, I take this as asking:

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do. Heard about DNL and INL, offset and gain errors? It would be a great thing to get rid of this mess, but they keep coming up.

(And yes, you can not gain DR by oversampling, but you can improve the SNR and the number of effective bits, this is how all sigma-delta converters are working)

This is my other favorite:

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

Where do you come from? Don't you have education there? I'm really interested, how can people end up with such an incredibly stupid conclusion...

[ankorwatt]

DxOMarks are useless and only made to entertain hobby photographers.

remarks like this just shows great ignorance, learn to understand DXO measurement data and you would not write  posts like this

[tnargs]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page, which is easily accesed from the About tab.  Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance."  Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure.  But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

well spotted! Half way down the same page says "So before comparing cameras with Sensor Overall Score, it is important to first determine the resolution you are looking for (which largely depends on the size of the screen or the print you intend to use or produce). Once you choose an appropriate resolution, the Sensor Overall Score becomes a fair and powerful tool with which to make comparisons." My memory was pretty good from a couple of years back!

@tnargs, if anything, that just makes the main point of post #1 even stronger. 

yes, my point

[tnargs]

....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

[neuroanatomist]

What has Canon sales to do with DXO ?

Nothing.  Which is exactly my conclusion:

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

If Consumer Reports gives one product a much higher rating that another product, that usually has a tangible impact on sales, i.e. those reports impact buying decisions.  DxOMark has given Nikon higher ratings than Canon for years, and there does not appear to have been any impact of that on sales, i.e. they have no impact on buying decisions. 

Given that:
a) there are flaws and ambiguity in their scores, not to mention some apparently aberrant results (e.g. 70-200 II),
b) their scores apparently (according to them) should not be used to compare cameras of different resolutions,
c) they are scoring only sensor performance, which is just one part of camera performance,
and d) the reduction of a complex imaging system to a single number is essentially meaningless anyway,

I would argue that not only do DxO's Scores have no impact on buying decisions in aggregte, they should have only a minor impact, if any at all, on personal buying decisions. 

As I've pointed out before, their measurements are usually quite good (I say usually because of the above-referenced 70-200 II issue, where DxOMark are the only ones in the world who seem to think the MkI is better than the MkII, and chose not to test another copy but rather to defend their results, which really doesn't help their credibility).  But their Scores are neither meaningful nor useful.