DxOMark vs. Reality

[Hillsilly]

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

And I think that's where the problem lies.  Very few people care or want to know about the technical side.  They just want to take great photos.  To do this you need a camera and if you're doing some research you will come across the DxO site.  How awesome is this!!  Not only do they rank all of the cameras but you can also do a comparison between them.  Despite disclaimers and fine print, you really get the impression that the DxO Mark is an objective assessment of one cameras overall ability vs another.

Anyways, I'm only anti-DxO because my camera gets a lowly "73".  Whereas my Nikon D5100 wielding sister gets an "80".  Yet my camera is sooooo much better....

[aj1575]

Actually, there is only one mistake that DxOMark makes. They should drop the sensor ranking / rating. They do a great job with all the measurements and analysis, and the graphs are very informative. But the rating does not make sense; even the new one, where they try to differentiate between portrait, landscape and high ISO does not really make sense.
Just leave the graphs there, and people can find their own conclusion.

[elflord]

Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs do not perform well at high ISOs. A "general" rating does not work with a special purpose tool.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera. 

I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same". The 7D is substantially better than the 10D for example (9 points in the overall score,  dynamic range substantiially better)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

[Canon-F1]

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.   

[straub]

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

[Maui5150]

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices 

This +1000

Most people, especially Professionals in a field, are not fools.

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

In fact, if DxOMarks is to be trusted, then their should be screams that PhaseOne are TOTAL PIECES OF CRAP and that NIKON is ripping off people by selling any camera above $3000, because the D800 is the ONLY CAMERA anyone should buy willing to spend more than $3000 and those under should go for the D600. 

Not saying they are not good cameras, but what the last year plus has highlighted to me, DxOMarks Scores are COMPLETELY IRRELEVANT AND MEANINGLESS.

An erroneous equation provides erroneous results

[straub]

what are DXO´s some would ask.   

Well, they have the word "Science" in their logo and Reichmann doesn't :-)

[Maui5150]

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 

Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations.  So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are.  Anyone can produce DATA.  But data is not information. 

What DxOMarks lacks is RELEVANCE

[M.ST]

DxOMarks are useless and only made to entertain hobby photographers.

[davidpeter]

Ok, I take this as asking:

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do. Heard about DNL and INL, offset and gain errors? It would be a great thing to get rid of this mess, but they keep coming up.

(And yes, you can not gain DR by oversampling, but you can improve the SNR and the number of effective bits, this is how all sigma-delta converters are working)

This is my other favorite:

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

Where do you come from? Don't you have education there? I'm really interested, how can people end up with such an incredibly stupid conclusion...

[tnargs]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page, which is easily accesed from the About tab.  Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance."  Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure.  But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

well spotted! Half way down the same page says "So before comparing cameras with Sensor Overall Score, it is important to first determine the resolution you are looking for (which largely depends on the size of the screen or the print you intend to use or produce). Once you choose an appropriate resolution, the Sensor Overall Score becomes a fair and powerful tool with which to make comparisons." My memory was pretty good from a couple of years back!

@tnargs, if anything, that just makes the main point of post #1 even stronger. 

yes, my point

[tnargs]

....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

[neuroanatomist]

What has Canon sales to do with DXO ?

Nothing.  Which is exactly my conclusion:

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

If Consumer Reports gives one product a much higher rating that another product, that usually has a tangible impact on sales, i.e. those reports impact buying decisions.  DxOMark has given Nikon higher ratings than Canon for years, and there does not appear to have been any impact of that on sales, i.e. they have no impact on buying decisions. 

Given that:
a) there are flaws and ambiguity in their scores, not to mention some apparently aberrant results (e.g. 70-200 II),
b) their scores apparently (according to them) should not be used to compare cameras of different resolutions,
c) they are scoring only sensor performance, which is just one part of camera performance,
and d) the reduction of a complex imaging system to a single number is essentially meaningless anyway,

I would argue that not only do DxO's Scores have no impact on buying decisions in aggregte, they should have only a minor impact, if any at all, on personal buying decisions. 

As I've pointed out before, their measurements are usually quite good (I say usually because of the above-referenced 70-200 II issue, where DxOMark are the only ones in the world who seem to think the MkI is better than the MkII, and chose not to test another copy but rather to defend their results, which really doesn't help their credibility).  But their Scores are neither meaningful nor useful.

[PackLight]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.

Interesting you said 200mm, before I made that post it was the first lens I did look at. At f/2 the Nikon version is very bad in the corners. Supertele's are designed to be used wide open, wide open the Canon wins.

Since Canon doesn't have a 200-400mm yet, Nikon wins by default.

I compared the top Nikon and Canon 70-200mm before I made the last post, because I knew it is where you would want to go next. Again, in the side by side the Canon wins noticeably. 

The 24mm f/1.4 versions of Canon and Nikon, Nikon is near a match but in no way wins. Even wider the 14mm f/.8 the Nikon lens is much softer. I would give Nikon the edge for wide angle, not because of its lenses but its sensor.

If you are comparing top of the line lenses to top of the line lenses Nikon doesn't win. In some categories it gets close to matching but not wining.

[straub]

I took this 5 min ago to show the difference between one of  my 5dmk2 and d800

Congratulations, you are the first one to notice that D800 has more DR than 5D2 at base ISO. First prize is a set of tickets for Titanic on it's maiden voyage across the Atlantic.