October 24, 2014, 08:45:01 PM

Author Topic: DxOMark vs. Reality  (Read 81098 times)

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #60 on: October 10, 2012, 06:22:45 PM »
Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

I understand that there are people in this forum who repeat the same thing over and over -- that doesn't mean that they have "proven" anything, or even that they are correct.

There are at least two fairly obvious reasons why the DxO mark number is not flawed.

First, the choice of normalization baseline is arbitrary. When comparing (for example) an 21mpx and a 36mpx camera, you could normalize to 21mpx instead and get equivalent (but lower) . You won't hit the "quantization ceiling" if you do this.

Second, even if quantization error is greater than actual sensor error at the black point,the shadow noise level a few EV above the blackpoint will exceed the quantization error. So the bump from 14EV to 14.4EV will manifest as cleaner shadows and  an increase in usable dynamic range.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #60 on: October 10, 2012, 06:22:45 PM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #61 on: October 10, 2012, 06:28:46 PM »
Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) . 

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.

Meh

  • 5D Mark III
  • ******
  • Posts: 700
    • View Profile
Re: DxOMark vs. Reality
« Reply #62 on: October 10, 2012, 06:41:05 PM »
Oh boy, what a thread.

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #63 on: October 10, 2012, 06:52:06 PM »
What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.    ;D

Well, for a start, they run what is by any reasonable measure the leading sensor benchmark site.

They also implemented raw processing software.

How many industry leading benchmarks and raw converters do their critics on camera rumors sites publish ?

Many seems to have problems to understand exposure, the subject's DR and the cameras DR when two cameras are compared with the same parameters,   and  also the benefit of a large DR . Throwing sh... on DXO because of DXOs Canon sensor ratings only shows lack of knowledge.

Sure it is important to know if certain cameras have a higher DR, better color or whatever, but none of them are the point in what is wrong with DxO's testing. It is how they arrive at the final number that is flawed and how they present this number to the public.

It has been a while since I consulted DxO and the reviews they put out. The reason is that while they do some reasonable tests that can be informative they make assumptions that the components they feel are important can give a camera an overall rating.  The assume Color Depth is Portrait, DR is landscape and Sports is low ISO. All are important components, and testing individually is good.

If your new to the DSLR world and are trying to learn you flip to the page that shows the D800, it shows a rating of 95. 
You have to take the time to read the fine print to find out that the rating number, as you put it, is just sh.... that DxO is throwing out. There is no standard for combining these three test to arrive at a number, other than the standard DxO has come up with.

well_dunno

  • Canon 6D
  • *****
  • Posts: 356
    • View Profile
Re: DxOMark vs. Reality
« Reply #64 on: October 10, 2012, 06:52:20 PM »
hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
-  They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors  in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

Cheers!

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #65 on: October 10, 2012, 06:56:03 PM »

So, this discussion, which one has mounted the biggest d1ck... sorry, sensor, in a camera case, is total bu11shit. I think, none of you has ever come into a situation, when he thought stuff like "crap, the dynamic range in this picture is too low. The customer will be very unpleased. I wish, i had a Nikon."


Yes and No

No customer as I am not a Pro Photog

And

Yes, many times a higher DR would have been nice.

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #66 on: October 10, 2012, 07:05:01 PM »
hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
-  They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors  in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

See my post elsewhere in this thread.

http://www.canonrumors.com/forum/index.php?topic=10020.msg181164#msg181164

The sensor is an important piece, but it isn't everything. (essentially (2) on your list). I started with Canon because I knew other Canon users. Even then, I was familiar with review sites like snapsort that take DxO scores into account, so Canon needed to deliver a respectable performance.

When I went to upgrade, I could have switched (I had a rebel and two non-L lenses). However, for my application, high ISO performance is much more important than dynamic range at base ISO. So there's a very substantial benefit to having a full frame camera, but as far as I'm concerned, a D800 would not be any better than a 5DIII (indeed, not noticably better than my 5DII) I also think Canon have some compelling advantages over other brands (see my other post)



canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #66 on: October 10, 2012, 07:05:01 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14750
    • View Profile
Re: DxOMark vs. Reality
« Reply #67 on: October 10, 2012, 07:09:56 PM »
give one example there Canons sensor is better

Ok, you like wide DR.  I need to shoot at high ISO.  If I need to shoot at ISO 12800, which sensor offers a wider DR - the D800 or the 5DIII?

do you   understand the benefit and  the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .

Yes, it's one of several ways to extract the maximum amount of information from a RAW file.  But the bottom line is that you can't extract more information than is there to begin with.

Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) . 

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.

Exactly.  Picking the Civic over the Corvette (was that analogy even in this thread?!?) makes sense if fuel economy or small parking spaces are your biggest needs, picking the Corvette over the Civic makes sense if you engage in street racing, and neither of them make sense if you need to haul your 2-ton yacht to the harbor.

Those of you who defend DxO scores, why do you have Canon gear?

Excellent question! 

Maybe it's a nuance, but speaking for myself, I defend DxO's measurements (usually), but not their scores.  Their scores do not affect my buying decisions because I understand how they generate them (mostly, within the limits of their published methods, which lack some details), and I know how to interpret their underlying raw data for myself and draw my own conclusions based on them.  I also use DxO software for RAW conversions (and I'm pleased as punch that as of today it supports the 1D X, because now I can get away from the kludgy DPP user interface).

Some of the reasons I have Canon gear are that they have lenses that better meet my needs, I prefer the ergonomics of the Canon bodies and menus, and I detest the color yellow.
« Last Edit: October 10, 2012, 07:18:00 PM by neuroanatomist »
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Meh

  • 5D Mark III
  • ******
  • Posts: 700
    • View Profile
Re: DxOMark vs. Reality
« Reply #68 on: October 10, 2012, 07:19:01 PM »
I love Canon cameras and lenses.  There's a certain sexiness to Canon.  Nikon seems so utilitarian... they get the job done but I don't get the same feeling from it.  I need that feeling.  I don't get that feeling from DR at ISO100.  :D

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3934
    • View Profile
Re: DxOMark vs. Reality
« Reply #69 on: October 10, 2012, 08:18:24 PM »

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 


Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations.  So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are.  Anyone can produce DATA.  But data is not information. 

What DxOMarks lacks is RELEVANCE

You must be blind then if you can't see the real world difference the extra DR can make in real world properly exposed shots taken with D800 vs 5D3. It's plenty relevant for some shots.


LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3934
    • View Profile
Re: DxOMark vs. Reality
« Reply #70 on: October 10, 2012, 08:19:21 PM »
DxOMarks are useless and only made to entertain hobby photographers.

Yay! I love to run around bragging about how ignorant I am! Yay!

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3934
    • View Profile
Re: DxOMark vs. Reality
« Reply #71 on: October 10, 2012, 08:29:02 PM »
I do agree that it's best to look at their plots and RAW numbers and not there overall scores since they may not weigh various factors the way you would plus there isn't really any a way to look at so many different aspects at once and come up with one universal score to give meaning to it all at once.

I also agree that their lens testing appears to leave MUCH to be desired. Lens testing is done by different people there and it's much trickier to carry out well. I ignore their lens tests but their sensor data generally seems to be pretty solid.

OK maybe DxO tests have not crashed the market for Canon yet, but as you say they, and general more over the top crippling of silly little things, sure have made their user base restless as you can tell but peeking at any forum on the web. Switching systems is a big deal for many (it's a pain, costs some money, some don't like the Nikon UI as much, Nikon doesn't have 70-200 f/4 IS and such they are mostly all huge fast stuff or consumer stuff) and it can take a while for things to build up to make many switch, maybe they did not get there quite yet, but they could be heading there. Also the bulk of sales are probably in the Rebel market. Also the 5D2 was aided a lot by the video guys nabbing them like mad and Nikon hasn't had Exmor in FF smaller, more reasonably priced bodies until recently, etc.

And whoever markets things better often matters even more. Apple IIe and IBM PC were utter trash as was Windows for a long time (and even now to an extent) and yet that stuff sold and the far more advanced stuff, sadly, did not thanks to horrible management and a computer press often too easily bought and paid for.

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #72 on: October 10, 2012, 09:00:13 PM »
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #72 on: October 10, 2012, 09:00:13 PM »

Meh

  • 5D Mark III
  • ******
  • Posts: 700
    • View Profile
Re: DxOMark vs. Reality
« Reply #73 on: October 10, 2012, 10:12:39 PM »
Oh boy, what a thread.

yes and mybe some have learn something
1 there is noting like a right exposure
2.How you expose depends on the motive and the  motives DR and what camera you use
3.With a camera and 14 stop DR you can cover more from the topp to the bottom, and adjust with curves and other tools and  even mark an area and do a selektive adjustments to show more information from shadows  for example.

It's a nice thought you have that somebody might have learned something... but sadly the evidence suggests that somebody hasn't learned a thing.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14750
    • View Profile
Re: DxOMark vs. Reality
« Reply #74 on: October 10, 2012, 11:09:31 PM »
At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800  red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data? 
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #74 on: October 10, 2012, 11:09:31 PM »