DxOMark vs. Reality

[tnargs]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.

 

[dtaylor]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.

I would be curious to see that if you have a link. If true, they need to post this in big, bold type on every page.

 

[neuroanatomist]

IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page, which is easily accesed from the About tab. Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance." Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure. But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

@tnargs, if anything, that just makes the main point of post #1 even stronger. If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world. Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot.

BTW, putting it in big bold type on every page wouldn't help. They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?

 

[Hillsilly]

Some valid points above and its been interesting reading the discussions about DxOMark in recent months and gaining a better understanding of their tests. I used to find them very influential - A year or two ago, I was considering purchasing a little Pentax kit for travelling (DSLR and a couple of pancake lenses). But DxO gave their lenses such bad ratings that I gave up on the idea. And I've got almost a basic understanding of things. I don't know what complete beginners make of it all. Imagine how good Canon sales would be if they were competitve in DxO rankings.

 

[PackLight]

BTW, putting it in big bold type on every page wouldn't help. They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?

At least one, the guy that is always in front of me.

 

[lfg530]

See how empty this is.

Lets look at the top side of the line up. This will take it from talking about 1 lens to 10.
Hop over to TDP's ISO charts and compare any of the big supertele primes 200mm or up, version I or version II.
If you find a Nikon that outperforms any of its Canon's matches old version or new let me know. I haven't compared one yet that did.
A point to consider, the Nikon lenses ISO on the site were shot with the "superior" Nikon sensor.

Sure you can point out, the Canon lenses cost more. Canon knows there long lenses are better, they demand a premium and people pay it.

Which direction should we go next, I am sure there is a Nikon nugget somewhere. But, I know there are good lenses made for Nikon cameras. Just look up Zeiss.
[/quote]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.

 

[Otara]

Noone thinks it might just be that Canon is mostly selling 18MP sensors, and Nikon is mostly selling 16MP sensors at the low end of the DSLR market?

The market tends to go for simple differentiation rather than the more complex arguments Im seeing, particularly when there isnt an easier way to tell which one is really better.

Otara

 

[davidpeter]

DxO, here we go again.

I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

For the reasonable ones, they have well detailed protocols, and measurement methods:

http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements
http://www.dxo.com/us/embedded_imaging/scientific_publications

And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

But you have to understand, that this is a pixel based data (so when two sensors share the same SNR, the one with the bigger MP will give better IQ downsized to a given resolution). They try to compensate this in the scores, but score vs. charts is like photoshopped jpeg vs. RAW. The first may look better, but the second carries the more information.

I always smile, when see comments "DR tested by me". Yep, and how? Because that is the question. At DxO, we know that exactly.

But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.


Finally, let me have some observation on a comment, which made me writing this one:

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs are expensive, as they are:
Big in size. In Si technology, price grows exponentially with size. That's why they even glue the sensor form many pieces some times.
They are rare. Mass production would bring prices down to it's quarter.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests. And we already talked about the question of resolution.

So, i tried once more, I promise this was my last attempt on this forum, unless asked...

 

[weixing]

Hi,
IMHO, DxO comparison will be valid if RAW file is RAW data, but in reality RAW file is not RAW data especially Nikon RAW file. IMHO, DxO should be ok to compare camera models from same brand since most likely the way they handle the RAW data will be similar.

Have a nice day.

 

[dtaylor]

DxO, here we go again.

DxO defender, here we go again.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly.

Ergo DxO is silly because that is exactly what they claim can be done when they publish their scores. I will stop hammering them for their stupid overall scores when they remove them all from the site.

And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

I contest this point. At least as far as dynamic range is concerned, there are repeat instances where they are clearly wrong.

But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.

Are you part of DxO? Well...we will stop "wasting your time" when you stop wasting ours publishing silly overall scores that are repeated ad nauseam in forums. And when you correct some of the obvious flaws in your test results.

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs are expensive...

Yes, they are. Their IQ also wipes the floor with consumer APS-C equipment. (And that's coming from a huge fan of today's APS-C sensors!) And DxO looks stupid for claiming otherwise.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests.

If you work for DxO, do the entire team a favor: buy a transmission step wedge and use it. Don't run it through a flawed computer analysis. Actually use it and eyeball the output. You will be embarrassed at some of the mistakes in your database.

 

[Hillsilly]

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

And I think that's where the problem lies. Very few people care or want to know about the technical side. They just want to take great photos. To do this you need a camera and if you're doing some research you will come across the DxO site. How awesome is this!! Not only do they rank all of the cameras but you can also do a comparison between them. Despite disclaimers and fine print, you really get the impression that the DxO Mark is an objective assessment of one cameras overall ability vs another.

Anyways, I'm only anti-DxO because my camera gets a lowly "73". Whereas my Nikon D5100 wielding sister gets an "80". Yet my camera is sooooo much better....

 

[aj1575]

Actually, there is only one mistake that DxOMark makes. They should drop the sensor ranking / rating. They do a great job with all the measurements and analysis, and the graphs are very informative. But the rating does not make sense; even the new one, where they try to differentiate between portrait, landscape and high ISO does not really make sense.
Just leave the graphs there, and people can find their own conclusion.

 

[elflord]

Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs do not perform well at high ISOs. A "general" rating does not work with a special purpose tool.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera.

I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same". The 7D is substantially better than the 10D for example (9 points in the overall score, dynamic range substantiially better)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

 

[Canon-F1]

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask. ;D

 

[straub]

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

 

[Maui5150]

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices

This +1000

Most people, especially Professionals in a field, are not fools.

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

In fact, if DxOMarks is to be trusted, then their should be screams that PhaseOne are TOTAL PIECES OF CRAP and that NIKON is ripping off people by selling any camera above $3000, because the D800 is the ONLY CAMERA anyone should buy willing to spend more than $3000 and those under should go for the D600.

Not saying they are not good cameras, but what the last year plus has highlighted to me, DxOMarks Scores are COMPLETELY IRRELEVANT AND MEANINGLESS.

An erroneous equation provides erroneous results

 

[straub]

what are DXO´s some would ask. ;D

Well, they have the word "Science" in their logo and Reichmann doesn't

 

[Maui5150]

@tnargs, if anything, that just makes the main point of post #1 even stronger. If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world. Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot.

Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations. So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are. Anyone can produce DATA. But data is not information.

What DxOMarks lacks is RELEVANCE

 

[M.ST]

DxOMarks are useless and only made to entertain hobby photographers.

 

[davidpeter]

Ok, I take this as asking:

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do. Heard about DNL and INL, offset and gain errors? It would be a great thing to get rid of this mess, but they keep coming up.

(And yes, you can not gain DR by oversampling, but you can improve the SNR and the number of effective bits, this is how all sigma-delta converters are working)

This is my other favorite:

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

Where do you come from? Don't you have education there? I'm really interested, how can people end up with such an incredibly stupid conclusion...