DxOMark vs. Reality

[elflord]

Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

I understand that there are people in this forum who repeat the same thing over and over -- that doesn't mean that they have "proven" anything, or even that they are correct.

There are at least two fairly obvious reasons why the DxO mark number is not flawed.

First, the choice of normalization baseline is arbitrary. When comparing (for example) an 21mpx and a 36mpx camera, you could normalize to 21mpx instead and get equivalent (but lower) . You won't hit the "quantization ceiling" if you do this.

Second, even if quantization error is greater than actual sensor error at the black point,the shadow noise level a few EV above the blackpoint will exceed the quantization error. So the bump from 14EV to 14.4EV will manifest as cleaner shadows and an increase in usable dynamic range.

 

[elflord]

Oly OM-D, because DxOMark says it's better than a 7D anyway, right? : ) .

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.

 

[Meh]

Oh boy, what a thread.

 

[PackLight]

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask. ;D

Well, for a start, they run what is by any reasonable measure the leading sensor benchmark site.

They also implemented raw processing software.

How many industry leading benchmarks and raw converters do their critics on camera rumors sites publish ?

Many seems to have problems to understand exposure, the subject's DR and the cameras DR when two cameras are compared with the same parameters, and also the benefit of a large DR . Throwing sh... on DXO because of DXOs Canon sensor ratings only shows lack of knowledge.

Sure it is important to know if certain cameras have a higher DR, better color or whatever, but none of them are the point in what is wrong with DxO's testing. It is how they arrive at the final number that is flawed and how they present this number to the public.

It has been a while since I consulted DxO and the reviews they put out. The reason is that while they do some reasonable tests that can be informative they make assumptions that the components they feel are important can give a camera an overall rating. The assume Color Depth is Portrait, DR is landscape and Sports is low ISO. All are important components, and testing individually is good.

If your new to the DSLR world and are trying to learn you flip to the page that shows the D800, it shows a rating of 95.
You have to take the time to read the fine print to find out that the rating number, as you put it, is just sh.... that DxO is throwing out. There is no standard for combining these three test to arrive at a number, other than the standard DxO has come up with.

 

[well_dunno]

hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
- They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

Cheers!

 

[PackLight]

So, this discussion, which one has mounted the biggest d1ck... sorry, sensor, in a camera case, is total bu11shit. I think, none of you has ever come into a situation, when he thought stuff like "crap, the dynamic range in this picture is too low. The customer will be very unpleased. I wish, i had a Nikon."

Yes and No

No customer as I am not a Pro Photog

And

Yes, many times a higher DR would have been nice.

 

[elflord]

hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
- They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

See my post elsewhere in this thread.

http://www.canonrumors.com/forum/index.php?topic=10020.msg181164#msg181164

The sensor is an important piece, but it isn't everything. (essentially (2) on your list). I started with Canon because I knew other Canon users. Even then, I was familiar with review sites like snapsort that take DxO scores into account, so Canon needed to deliver a respectable performance.

When I went to upgrade, I could have switched (I had a rebel and two non-L lenses). However, for my application, high ISO performance is much more important than dynamic range at base ISO. So there's a very substantial benefit to having a full frame camera, but as far as I'm concerned, a D800 would not be any better than a 5DIII (indeed, not noticably better than my 5DII) I also think Canon have some compelling advantages over other brands (see my other post)

 

[Loswr]

give one example there Canons sensor is better

Ok, you like wide DR. I need to shoot at high ISO. If I need to shoot at ISO 12800, which sensor offers a wider DR - the D800 or the 5DIII?

do you understand the benefit and the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .

Yes, it's one of several ways to extract the maximum amount of information from a RAW file. But the bottom line is that you can't extract more information than is there to begin with.

Oly OM-D, because DxOMark says it's better than a 7D anyway, right? : ) .

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.

Exactly. Picking the Civic over the Corvette (was that analogy even in this thread?!?) makes sense if fuel economy or small parking spaces are your biggest needs, picking the Corvette over the Civic makes sense if you engage in street racing, and neither of them make sense if you need to haul your 2-ton yacht to the harbor.

Those of you who defend DxO scores, why do you have Canon gear?

Excellent question!

Maybe it's a nuance, but speaking for myself, I defend DxO's measurements (usually), but not their scores. Their scores do not affect my buying decisions because I understand how they generate them (mostly, within the limits of their published methods, which lack some details), and I know how to interpret their underlying raw data for myself and draw my own conclusions based on them. I also use DxO software for RAW conversions (and I'm pleased as punch that as of today it supports the 1D X, because now I can get away from the kludgy DPP user interface).

Some of the reasons I have Canon gear are that they have lenses that better meet my needs, I prefer the ergonomics of the Canon bodies and menus, and I detest the color yellow.

 

[Meh]

I love Canon cameras and lenses. There's a certain sexiness to Canon. Nikon seems so utilitarian... they get the job done but I don't get the same feeling from it. I need that feeling. I don't get that feeling from DR at ISO100.

 

[LetTheRightLensIn]

@tnargs, if anything, that just makes the main point of post #1 even stronger. If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world. Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot.

Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations. So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are. Anyone can produce DATA. But data is not information.

What DxOMarks lacks is RELEVANCE

You must be blind then if you can't see the real world difference the extra DR can make in real world properly exposed shots taken with D800 vs 5D3. It's plenty relevant for some shots.

 

[LetTheRightLensIn]

DxOMarks are useless and only made to entertain hobby photographers.

Yay! I love to run around bragging about how ignorant I am! Yay!

 

[LetTheRightLensIn]

I do agree that it's best to look at their plots and RAW numbers and not there overall scores since they may not weigh various factors the way you would plus there isn't really any a way to look at so many different aspects at once and come up with one universal score to give meaning to it all at once.

I also agree that their lens testing appears to leave MUCH to be desired. Lens testing is done by different people there and it's much trickier to carry out well. I ignore their lens tests but their sensor data generally seems to be pretty solid.

OK maybe DxO tests have not crashed the market for Canon yet, but as you say they, and general more over the top crippling of silly little things, sure have made their user base restless as you can tell but peeking at any forum on the web. Switching systems is a big deal for many (it's a pain, costs some money, some don't like the Nikon UI as much, Nikon doesn't have 70-200 f/4 IS and such they are mostly all huge fast stuff or consumer stuff) and it can take a while for things to build up to make many switch, maybe they did not get there quite yet, but they could be heading there. Also the bulk of sales are probably in the Rebel market. Also the 5D2 was aided a lot by the video guys nabbing them like mad and Nikon hasn't had Exmor in FF smaller, more reasonably priced bodies until recently, etc.

And whoever markets things better often matters even more. Apple IIe and IBM PC were utter trash as was Windows for a long time (and even now to an extent) and yet that stuff sold and the far more advanced stuff, sadly, did not thanks to horrible management and a computer press often too easily bought and paid for.

 

[PackLight]

This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images lifted in the same way in photoshop so we should be able to see country landscape.

DR

Sorry but this set is bull. Not a believable comparison at all.

 

[Meh]

Oh boy, what a thread.

yes and mybe some have learn something
1 there is noting like a right exposure
2.How you expose depends on the motive and the motives DR and what camera you use
3.With a camera and 14 stop DR you can cover more from the topp to the bottom, and adjust with curves and other tools and even mark an area and do a selektive adjustments to show more information from shadows for example.

It's a nice thought you have that somebody might have learned something... but sadly the evidence suggests that somebody hasn't learned a thing.

 

[Loswr]

At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800 red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data?

 

[Loswr]

give one example there Canons sensor is better

Granted, the difference above is pretty small. So, while we're at it, let's compare the Canon 7D with it's contemporary competition, the D300s (released about one month apart). Which has better DR through almost the entire ISO range, according to the almighty DxO's scoring and normalized data?

 

[tnargs]

This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images lifted in the same way in photoshop so we should be able to see country landscape.

DR

Sorry but this set is bull. Not a believable comparison at all.

Makes one wonder if he is just a troll.... posts are exclusively anti-Canon pro-Nikon (Sony type N)

 

[dtaylor]

Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera.

You know darn well their scores do not reflect how good the camera is as a "general purpose camera."

I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras. When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede. When the same "fan" is confronted with the fact that the score for a consumer DSLR is higher than a 40 MP MF back, all of a sudden the score isn't a score of overall IQ, and nobody is using those scores to begin with, and stop tossing out red herrings.

Which is it?

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?
[/quote]

Visit his website.

 

[dtaylor]

Ok, I take this as asking:

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do.

I love how the excuse for DxO's results keeps changing. It's due to down scaling. No, it's due to non-linear sensors / amps. No, it's due to... You guys need to get together and compare your notes :

The sensors are not designed to expand DR through non-linear behavior. They are, for the purposes of this discussion, linear. Further more, they have noise. And like all analog to digital systems their range is actually less than on paper because of things like noise.

If they were absolutely perfect in every way you could expect a maximum of 14 stops. They're not.

Face it: DxO botched another test.

Now if you'll excuse me, I have to order my Nikon D3200 through Amazon since it's better than a Hasselblad. Talk about a steal ;D

 

[dtaylor]

If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.