DxOMark vs. Reality

[elflord]

I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when  there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.

[tron]

I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when  there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.

Thanks!

[nightbreath]

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.

I didn't quite catch this. Could you expand your answer on my example? Here's how I understand downsampling:

Assume that we have 4 neighboring pixels that represent in reality 4 black squares, but in our RAW file (due to noise impact) they have next brightness levels:
1 0
0 3

So, when I downsample I get 1 pixel that has brightness level 1 (for example, if I downsample by next formula: (1+0+0+3)/4 ).

It means that before downsampling we were looking at image with average noise level equal to 1 and we do the same after downsampling.

So what am I doing wrong compared to all of you that have noise level reduced after downsammpling?

P.S. Sorry for asking silly questions, i'd just like to understand 

[BXL]

Those of you who defend DxO scores, why do you have Canon gear?

Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks"

[heptagon]

Those of you who defend DxO scores, why do you have Canon gear?

Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks"

You wouldn't go offroad with a racecar or wear slick tires in the mud.

Every job needs the right tool.

[elflord]

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.

I didn't quite catch this. Could you expand your answer on my example? Here's how I understand downsampling:

Assume that we have 4 neighboring pixels that represent in reality 4 black squares, but in our RAW file (due to noise impact) they have next brightness levels:
1 0
0 3

So, when I downsample I get 1 pixel that has brightness level 1 (for example, if I downsample by next formula: (1+0+0+3)/4 ).

It means that before downsampling we were looking at image with average noise level equal to 1 and we do the same after downsampling.

So what am I doing wrong compared to all of you that have noise level reduced after downsammpling?

P.S. Sorry for asking silly questions, i'd just like to understand 

There are a couple of issues with the above example. First, the right measure for error is RMS, not arithmetic mean (that is, if you take two measurements, +1 and -1 for a variable whose true value is 0, the "error" is certainly not 0). 
In your example, the RMS error is

sqrt( 1/4( 3² + 1² + 0² + 0²) ) = 1.6.

 After you downsample, you get an RMS error of 1 -- that is, sqrt( (1-0)²/ 1 ).

Second the example is a bit unusual because you generally don't expect the error terms to be entirely of the same sign, and by averaging the signal the positive and negative errors cancel each other out.

If you have (for example) a "true" value of 10 and readings 6,3,9,17 (these I just randomly sampled from a normal distribution with mean 10, standard deviation 5).  We average these and we get 9 (an error of 1), whereas before we had an error of 5 -- 

sqrt[  ((10-6)²+(10-3)²)+(10-9)²+(10-17)²))/4  ].

In general, the expected value of our error drops by a factor of sqrt(N) -- so in my above example, the expected error in the beginning is the standard deviation of our distribution, 5. After we average four pixels, our expected error is 2.5  (5/sqrt(4)).

[well_dunno]

  It was more of a rhetorical question, which was the last line of my post earlier in this thread...

Cheers!

[nightbreath]

There are a couple of issues with the above example. First, the right measure for error is RMS, not arithmetic mean (that is, if you take two measurements, +1 and -1 for a variable whose true value is 0, the "error" is certainly not 0). 
In your example, the RMS error is

sqrt( 1/4( 3² + 1² + 0² + 0²) ) = 1.6.

 After you downsample, you get an RMS error of 1 -- that is, sqrt( (1-0)²/ 1 ).

Second the example is a bit unusual because you generally don't expect the error terms to be entirely of the same sign, and by averaging the signal the positive and negative errors cancel each other out.

If you have (for example) a "true" value of 10 and readings 6,3,9,17 (these I just randomly sampled from a normal distribution with mean 10, standard deviation 5).  We average these and we get 9 (an error of 1), whereas before we had an error of 5 -- 

sqrt[  ((10-6)²+(10-3)²)+(10-9)²+(10-17)²))/4  ].

In general, the expected value of our error drops by a factor of sqrt(N) -- so in my above example, the expected error in the beginning is the standard deviation of our distribution, 5. After we average four pixels, our expected error is 2.5  (5/sqrt(4)).

Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.

[bdunbar79]

Those of you who defend DxO scores, why do you have Canon gear?

Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks"

    There are different brand fanatic  and the arguments regarding why something suddenly not important anymore (when "their" brand suddenly is not the best ) and this  might take quite comical proportions ... Some very active writers, among others  here at Canonrumors's  are direct rabid - this despite the notice that they are intelligent enough to see the connections, there is an instinctive religious barrier that prevents them to see what's right in front of their eyes ... :-)

Oh the irony......................

[birtembuk]

Those of you who defend DxO scores, why do you have Canon gear?

Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks"

    There are different brand fanatic  and the arguments regarding why something suddenly not important anymore (when "their" brand suddenly is not the best ) and this  might take quite comical proportions ... Some very active writers, among others  here at Canonrumors's  are direct rabid - this despite the notice that they are intelligent enough to see the connections, there is an instinctive religious barrier that prevents them to see what's right in front of their eyes ... :-)

Religion ? What are you talking about mate ? To the vast majority, the biggest size they'll go is the 23" diagonal of their computer screen. Who is printing billboards ? If the dxo ratings were saying that my Canon gear is better than the competition, I for one would not give a damn. Conversely, I don't give a damn with the opposite. If your purchase decision is based on dxo ratings, please be my guest. But maybe one day you'll have to dump all your gear because dxo found out that the iPhone has a much better sensor ! For sure, dxo won't make me change a micron from my path. Religion or not.       

[MarkII]

Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.

I think you should explain how you conclude this from Elfiord's post - it does not follow at all.

The down-sampled figures are a perfectly reasonable way of comparing sensors. They correspond to what you can achieve if you print or view your photograph out at the same physical size and print DPI. From the numbers, it looks as if DXO actually measure this after performing down-sampling, rather than just applying a theoretical perfect adjustment. This means that the down-sampled numbers inherently include any 'environmental parameters' in the result.

No one argued against downsampling when the comparison was 5DII to D700...

[nightbreath]

Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.

I think you should explain how you conclude this from Elfiord's post - it does not follow at all.

The down-sampled figures are a perfectly reasonable way of comparing sensors. They correspond to what you can achieve if you print or view your photograph out at the same physical size and print DPI. From the numbers, it looks as if DXO actually measure this after performing down-sampling, rather than just applying a theoretical perfect adjustment. This means that the down-sampled numbers inherently include any 'environmental parameters' in the result.

No one argued against downsampling when the comparison was 5DII to D700...

I was not brand-specific and I'm not defending 5D Mk II vs. D700 scorings. As well as others I didn't have a clue what to look at when I first got to the DXO web-site and wanted to choose my next camera. That's why I am kind of frustrated now, because I kno what to look for.

As for environmental parameters, I was referring to ADC bit depth, as it is clear that by downsampling you just create data that was not in the image before downsampling (more than 14 stops of data in 14 bit depth image). I.e. if your measurements include noise floor evaluation that gives different results when you compare current image with other images, you just won't make an impression of a trusted source.

Why not come up with an algorithm that measures the whole image, not each pixel individually? In this case you don't need to downscale at all.

P.S. Please correct me where I'm wrong 

[aj1575]

Please look at these graphs:

And now tell me, which sensor should get a better mark for high ISO performance?

[marekjoz]

Please look at these graphs:

And now tell me, which sensor should get a better mark for high ISO performance?

The Nikon's one?

[infared]

The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data.  Sometimes thier "data" is in line with the general consensus of my other respected sources, many times not. I do know that specific products from manufacturers can vary from item to item..but DxO mark does not present their information with that variation. They present their information as "fact".
 I do not waste my time with ANYTHING that DxO Mark has to say because I have found many of their "opinions" to be less than valuable. I feel that anyone with experience and intelligence would do the same.
There are just so many other consistently reliable sources out there to get my information from that increase my awareness and knowledge of photographic tools.