December 20, 2014, 01:55:32 AM

Author Topic: DxOMark vs. Reality  (Read 84498 times)

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 15213
    • View Profile
Re: DxOMark vs. Reality
« Reply #75 on: October 10, 2012, 11:09:42 PM »
give one example there Canons sensor is better

Granted, the difference above is pretty small.  So, while we're at it, let's compare the Canon 7D with it's contemporary competition, the D300s (released about one month apart).  Which has better DR through almost the entire ISO range, according to the almighty DxO's scoring and normalized data?
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #75 on: October 10, 2012, 11:09:42 PM »

tnargs

  • Rebel T5i
  • ****
  • Posts: 138
    • View Profile
Re: DxOMark vs. Reality
« Reply #76 on: October 11, 2012, 02:19:58 AM »
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.

Makes one wonder if he is just a troll.... posts are exclusively anti-Canon pro-Nikon (Sony type N)

dtaylor

  • 1D Mark IV
  • ******
  • Posts: 849
    • View Profile
Re: DxOMark vs. Reality
« Reply #77 on: October 11, 2012, 03:30:21 AM »
Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera.

You know darn well their scores do not reflect how good the camera is as a "general purpose camera."
Quote
I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras. When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede. When the same "fan" is confronted with the fact that the score for a consumer DSLR is higher than a 40 MP MF back, all of a sudden the score isn't a score of overall IQ, and nobody is using those scores to begin with, and stop tossing out red herrings.

Which is it?

Quote
This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

Quote
Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

Quote
For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?
[/quote]

Visit his website.

dtaylor

  • 1D Mark IV
  • ******
  • Posts: 849
    • View Profile
Re: DxOMark vs. Reality
« Reply #78 on: October 11, 2012, 03:39:27 AM »
Ok, I take this as asking:

Quote
Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do.

I love how the excuse for DxO's results keeps changing. It's due to down scaling. No, it's due to non-linear sensors / amps. No, it's due to... You guys need to get together and compare your notes   ::)

The sensors are not designed to expand DR through non-linear behavior. They are, for the purposes of this discussion, linear. Further more, they have noise. And like all analog to digital systems their range is actually less than on paper because of things like noise.

If they were absolutely perfect in every way you could expect a maximum of 14 stops. They're not.

Face it: DxO botched another test.

Now if you'll excuse me, I have to order my Nikon D3200 through Amazon since it's better than a Hasselblad. Talk about a steal  ;D

dtaylor

  • 1D Mark IV
  • ******
  • Posts: 849
    • View Profile
Re: DxOMark vs. Reality
« Reply #79 on: October 11, 2012, 03:49:55 AM »
If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.

MarkII

  • PowerShot G1 X II
  • ***
  • Posts: 65
    • View Profile
Re: DxOMark vs. Reality
« Reply #80 on: October 11, 2012, 04:54:02 AM »
If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.

Yes - but possibly not in the way I think you mean. The original RAW image is spatially oversampled (20MP or more) with respect to the final output resolution (8MP). The process of converting the original high-resolution image to the smaller image is downsampling.

The Wikipedia pages are helpful here, and explain this probably better than I can:

  http://en.wikipedia.org/wiki/Oversampling
  http://en.wikipedia.org/wiki/Downsampling

These also give an example of how this can be exploited to give more resolution (DR) than the actual converter (sensor pixel).

tron

  • 1D X
  • *******
  • Posts: 1923
    • View Profile
Re: DxOMark vs. Reality
« Reply #81 on: October 11, 2012, 06:36:04 AM »
do you   understand the benefit and  the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .
This is interesting! I have also read it somewhere in this forum before and I was puzzled about one thing.

So please allow me to ask something:

If the middle grey is moved down 1-4 levels and the highlights are moved above then don't we lose the
mid tones?


canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #81 on: October 11, 2012, 06:36:04 AM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #82 on: October 11, 2012, 06:40:02 AM »
Quote
It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras.

It is a red herring, because it is not a "flaw" in methodology, it is a limitation in using a single score to describe "overall" performance.

It does not follow at all that the measurements are in any way incorrect.

As a line of argument, it is, in essence, an ad-hominem -- an attempt to troll the data for "incorrect" results, and attack the source, instead of addressing the real issue (which is that Canon's sensors don't perform well). Besides being a morally reprihensible approach, it is a logical fallacy. It suffers from all the dangers inherent in data mining (that if you look for a given result over a large amount of data, you can usually find it, especially if you don't have a very good understanding of what you are looking at).

Quote
When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede.

No, the score represents some subjective weighting of different measurements. "Overall" image quality is also a subjective notion (what does "overall" mean ?)

The measurements however are cold hard factual data. And in the case of Canon vs Nikon, they show that Canon struggles with shadow noise at low ISO. If you often use the camera at ISO 800 or more, this may not matter a whole lot, but if you shoot at ISO100 all the time, it might. The results put forth by way of DxOs measurements (again specifically shadow noise) have been validated by other users/commentators, so no amount of carrying on about medium format and other red herrings will make this go away.

You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

Sure you can. Dynamic range is the space between saturation point and a reference SNR. If your reference SNR is 0db (SNR=1), then quantization will bite you. But if you use a different floor, it won't. In practice, SNR=1 isn't where usable dynamic range starts, so you do gain by going from "14" to "14.4". Of course there is a trade between "detail" (spatial resolution)  for noise reduction -- that's why they normalize to the same resolution, it's also why it doesn't make sense to compare a high resolution to low resolution camera without doing that. BTW, some of these medium format (e.g.  80mpx )  backs you are fond of would not look nearly as good if you insisted on using a per-pixel metric to measure their performance.

Also, see my other point -- if you compare two cameras one at 21mpx and one at 36mpx, you could normalize to 21mpx instead of 36mpx. The choice of "target resolution" does not matter, the important point is that everything is normalized to the same resolution.

Quote
Quote
You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

However you slice and dice it, if you look at the two graphs, it's not "about the same". It's 0.7 stops on the Y axis at base ISO, or about 2 stops on the x axis (meaning that I can crank ISO up two stops on the 7D and get the same DR). That's a very substantial technological bump.

Quote
Visit his website.

I did, and my question stands -- I saw information on what he has done as a photographer. Again, as a benchmarker, engineer, etc, what are they ?
« Last Edit: October 11, 2012, 07:01:22 AM by elflord »

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #83 on: October 11, 2012, 07:52:15 AM »
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.

Are they, explain why.
Both cameras equal exposed  time/f-stop / iso  so there will be no highligts clipping, both cameras equaly lifted in the shadows =visible results

1, You are showing two pictures that are JPG without any explanation of the full work flow you used to arrive at the results.
2, You modified the pictures in photoshop. No explanation of what format you used, or if you did any processing on the RAW files.
3, I have shot similar sunsets with the 5D II. The results you are proposing are not what I would reasonably expect to see.

Last, this appears to be an attempt to exaggerate results to try and show the DR of D800. The D800 was measured by DxO. DxO's own results were not as wide as you are trying to make out.

Granted if your point is that the D800 has wider DR, it does. DxO measured it at over 14, but you do not have 14 stops of usable DR. You have a sensor that was measured at the extremes to have 14 stops. In reality you have far less. The same is true of the 5D II. What was measured as at extremes for both to be a difference of 3 stops, if DxO had tested what a camera can actually do in real world situations the difference would be closer to 1 or 2 stops. Still this is a fair difference, but to keep beating the 14 stop drum and warping pictures to over exaggerate a point that most here would concede to forces one to question your points.

The first set of pictures you posted demonstrated the point, and now I wonder if they were accurate. I have seen the side by side comparisons from reasonable sources and saw no need to scrutinize closely your first example because the difference is there and has been well established by other sources. But on the second it makes me question and doubt your examples.
« Last Edit: October 11, 2012, 07:54:47 AM by PackLight »

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 15213
    • View Profile
Re: DxOMark vs. Reality
« Reply #84 on: October 11, 2012, 08:14:31 AM »
At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800  red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data?

select print mode or make d800 pixel number  as many as Canon

LOL.   You're really grasping at straws now...  Go look again, but more carefully this time.  I did select Print mode. 

I'll even make it easy for you by reposting it, along with the Screen mode where as you can see (or can you, with your blinders on?) that the 5DIII has a greater DR advantage at high ISO without normalization. 
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 944
    • View Profile
    • marekjoz @flickr
Re: DxOMark vs. Reality
« Reply #85 on: October 11, 2012, 08:44:48 AM »
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...
« Last Edit: October 11, 2012, 08:48:26 AM by marekjoz »
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

marinien

  • Rebel SL1
  • ***
  • Posts: 80
    • View Profile
Re: DxOMark vs. Reality
« Reply #86 on: October 11, 2012, 09:10:19 AM »
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.
7D | EF-S 17-55 | EF 100mm f/2.8 L IS Macro | 580EX II | Benro C3780T + Markins M20

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 944
    • View Profile
    • marekjoz @flickr
Re: DxOMark vs. Reality
« Reply #87 on: October 11, 2012, 09:14:23 AM »
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.

<sarcasm off> :D

Anyway - the bigger the area in either low or high iso part of the graph, the bigger the difference between them there and the simplest way to calculate this area are integrals when you know the math formulas. But anyway it was just to cheer you all up in this serious discussion...
« Last Edit: October 11, 2012, 09:20:30 AM by marekjoz »
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #87 on: October 11, 2012, 09:14:23 AM »

mystic_theory

  • Guest
Re: DxOMark vs. Reality
« Reply #88 on: October 11, 2012, 09:25:23 AM »
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality?  ;D

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #89 on: October 11, 2012, 09:44:35 AM »
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality?  ;D

No, it means that the 1D X tested well with the three tests that DxO uses to generate their make believe scores.


canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #89 on: October 11, 2012, 09:44:35 AM »