DxOMark vs. Reality

  • Thread starter Thread starter Loswr
  • Start date Start date
Status
Not open for further replies.
dtaylor said:
MarkII said:
If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.

Yes - but possibly not in the way I think you mean. The original RAW image is spatially oversampled (20MP or more) with respect to the final output resolution (8MP). The process of converting the original high-resolution image to the smaller image is downsampling.

The Wikipedia pages are helpful here, and explain this probably better than I can:

http://en.wikipedia.org/wiki/Oversampling
http://en.wikipedia.org/wiki/Downsampling

These also give an example of how this can be exploited to give more resolution (DR) than the actual converter (sensor pixel).
 
Upvote 0
ankorwatt said:
do you understand the benefit and the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .
This is interesting! I have also read it somewhere in this forum before and I was puzzled about one thing.

So please allow me to ask something:

If the middle grey is moved down 1-4 levels and the highlights are moved above then don't we lose the
mid tones?
 
Upvote 0
It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras.

It is a red herring, because it is not a "flaw" in methodology, it is a limitation in using a single score to describe "overall" performance.

It does not follow at all that the measurements are in any way incorrect.

As a line of argument, it is, in essence, an ad-hominem -- an attempt to troll the data for "incorrect" results, and attack the source, instead of addressing the real issue (which is that Canon's sensors don't perform well). Besides being a morally reprihensible approach, it is a logical fallacy. It suffers from all the dangers inherent in data mining (that if you look for a given result over a large amount of data, you can usually find it, especially if you don't have a very good understanding of what you are looking at).

When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede.

No, the score represents some subjective weighting of different measurements. "Overall" image quality is also a subjective notion (what does "overall" mean ?)

The measurements however are cold hard factual data. And in the case of Canon vs Nikon, they show that Canon struggles with shadow noise at low ISO. If you often use the camera at ISO 800 or more, this may not matter a whole lot, but if you shoot at ISO100 all the time, it might. The results put forth by way of DxOs measurements (again specifically shadow noise) have been validated by other users/commentators, so no amount of carrying on about medium format and other red herrings will make this go away.

dtaylor said:
You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

Sure you can. Dynamic range is the space between saturation point and a reference SNR. If your reference SNR is 0db (SNR=1), then quantization will bite you. But if you use a different floor, it won't. In practice, SNR=1 isn't where usable dynamic range starts, so you do gain by going from "14" to "14.4". Of course there is a trade between "detail" (spatial resolution) for noise reduction -- that's why they normalize to the same resolution, it's also why it doesn't make sense to compare a high resolution to low resolution camera without doing that. BTW, some of these medium format (e.g. 80mpx ) backs you are fond of would not look nearly as good if you insisted on using a per-pixel metric to measure their performance.

Also, see my other point -- if you compare two cameras one at 21mpx and one at 36mpx, you could normalize to 21mpx instead of 36mpx. The choice of "target resolution" does not matter, the important point is that everything is normalized to the same resolution.

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

However you slice and dice it, if you look at the two graphs, it's not "about the same". It's 0.7 stops on the Y axis at base ISO, or about 2 stops on the x axis (meaning that I can crank ISO up two stops on the 7D and get the same DR). That's a very substantial technological bump.

Visit his website.

I did, and my question stands -- I saw information on what he has done as a photographer. Again, as a benchmarker, engineer, etc, what are they ?
 
Upvote 0
ankorwatt said:
PackLight said:
ankorwatt said:
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images lifted in the same way in photoshop so we should be able to see country landscape.

DR

Sorry but this set is bull. Not a believable comparison at all.

Are they, explain why.
Both cameras equal exposed time/f-stop / iso so there will be no highligts clipping, both cameras equaly lifted in the shadows =visible results

1, You are showing two pictures that are JPG without any explanation of the full work flow you used to arrive at the results.
2, You modified the pictures in photoshop. No explanation of what format you used, or if you did any processing on the RAW files.
3, I have shot similar sunsets with the 5D II. The results you are proposing are not what I would reasonably expect to see.

Last, this appears to be an attempt to exaggerate results to try and show the DR of D800. The D800 was measured by DxO. DxO's own results were not as wide as you are trying to make out.

Granted if your point is that the D800 has wider DR, it does. DxO measured it at over 14, but you do not have 14 stops of usable DR. You have a sensor that was measured at the extremes to have 14 stops. In reality you have far less. The same is true of the 5D II. What was measured as at extremes for both to be a difference of 3 stops, if DxO had tested what a camera can actually do in real world situations the difference would be closer to 1 or 2 stops. Still this is a fair difference, but to keep beating the 14 stop drum and warping pictures to over exaggerate a point that most here would concede to forces one to question your points.

The first set of pictures you posted demonstrated the point, and now I wonder if they were accurate. I have seen the side by side comparisons from reasonable sources and saw no need to scrutinize closely your first example because the difference is there and has been well established by other sources. But on the second it makes me question and doubt your examples.
 
Upvote 0
ankorwatt said:
neuroanatomist said:
ankorwatt said:
At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800 red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data?

select print mode or make d800 pixel number as many as Canon

LOL. You're really grasping at straws now... Go look again, but more carefully this time. I did select Print mode.

I'll even make it easy for you by reposting it, along with the Screen mode where as you can see (or can you, with your blinders on?) that the 5DIII has a greater DR advantage at high ISO without normalization.
 

Attachments

  • This one is Print.png
    This one is Print.png
    36.4 KB · Views: 738
  • THIS one is Screen.png
    THIS one is Screen.png
    36.2 KB · Views: 843
Upvote 0
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...
 
Upvote 0
marekjoz said:
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.
 
Upvote 0
marinien said:
marekjoz said:
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.

<sarcasm off> :D

Anyway - the bigger the area in either low or high iso part of the graph, the bigger the difference between them there and the simplest way to calculate this area are integrals when you know the math formulas. But anyway it was just to cheer you all up in this serious discussion...
 
Upvote 0
mystic_theory said:
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality? ;D

No, it means that the 1D X tested well with the three tests that DxO uses to generate their make believe scores.
 
Upvote 0
ankorwatt said:
Alrik89 said:
Guys, you've got a real problem, when you assess camera sensors basing on some stupid numbers like DxO.

If you're a professional photographer (a skilled one) then none of your customers will ever see the difference in the IQ/dynamic range/whatever between a Nikon and a Canon DSLR.

So, this discussion, which one has mounted the biggest d1ck... sorry, sensor, in a camera case, is total bu11shit. I think, none of you has ever come into a situation, when he thought stuff like "crap, the dynamic range in this picture is too low. The customer will be very unpleased. I wish, i had a Nikon."

Get a life.

A skilled photographer is not the same as he or she is a good copyist and know how a digital work flou works. Have you anytime worked in a darkroom with a negative film then you will understand what I mean and likewise it is with a raw files.

A skilled photographer with also skills in the digital "dark room" kan do so much more than a photographer with no photoshops skills. And even more with a camera who has 14 stops DR.
best regards , pro since 30 years back.

Well, when you need 14 stops DR to "rescue" your pictures via Photoshop, then you should ask yourself, if you`re a real pro.
When it comes to the the point where you really need 14 stops of DR, you've taken your picture in a wrong way.

And: none will see the difference. Neither a customer nor other pro photographer.
 
Upvote 0
marekjoz said:
marinien said:
marekjoz said:
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.

<sarcasm off> :D

Anyway - the bigger the area in either low or high iso part of the graph, the bigger the difference between them there and the simplest way to calculate this area are integrals when you know the math formulas. But anyway it was just to cheer you all up in this serious discussion...

Haha, thanks ;)
 
Upvote 0
ankorwatt said:
mystic_theory said:
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality? ;D

This will be very intresting to know, Canon have increased the DR and shows very good high iso scores
if the figures are OK ( are DXO still crap?)

Yes, the assumptions and overall score they draw from the tests are still crap. There scores are drawn from to narrow of a parameter to be of real use.
 
Upvote 0
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml

Any thoughts on this guys?
 
Upvote 0
shinyknights said:
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml

Any thoughts on this guys?

I think if you use DxO information it is important to know in what context they test, the test reults and ignore the values they place as scores.

As for the test at ISO 12800 you mention, as far as I am concerned the samples would be irrelevant as well because I wouldn't go that far with the 5D. I would with the 1D X. I think a comparison at ISO 6400, an ISO I might go to on occasion tells more as does the ISO 3200 comparison.
 
Upvote 0
shinyknights said:
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml

Any thoughts on this guys?

They have certain criteria like 30db signal to noise ratio ... any shot that gets below this is tossed out. If they change their criteria to (taking an arbitrary number) 25db, then perhaps 5d3 will have 11000 ISO (another arbitrary number) and D800 will have 4000 ISO (Yet another arbitrary number)... shifting the criteria around will swing the score back to the Canon Camp. Thats what pissed people about them is the criteria they use to weigh qualities of a sensor for it's "Overall Score".

The criteria they use effectively nullifies Canon's real world high ISO advantage.

The other criteria they have in place is that at the 30db SNR level, at least 18 bit color depth be mantained for the ISO score, now who plucked this number from someones... expletive is another debate altogether, but it again weights the balance in favor of the Sonikon sensors... (The color bit depth rears it's ugly head again for overall scores.)

I think even with current technology levels fo Canon sensors, with their high ISO advantage, DxO could tweak their criteria and Canon scores could be above Nikons... but there is one area where playing fuzzy math will not help canon and that is the DR number... this is what many Canon Loyaists want Canon to improve.
 
Upvote 0
shinyknights said:
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

Their scores are based on measurements, not picture samples.

If you want the full story behind the scores, click on the measurements tab. The scores compress a lot of information using subjective (and therefore somewhat arbitrary) criteria.
 
Upvote 0
MarkII said:
Yes - but possibly not in the way I think you mean. The original RAW image is spatially oversampled (20MP or more) with respect to the final output resolution (8MP). The process of converting the original high-resolution image to the smaller image is downsampling.

The Wikipedia pages are helpful here, and explain this probably better than I can:

http://en.wikipedia.org/wiki/Oversampling
http://en.wikipedia.org/wiki/Downsampling

These also give an example of how this can be exploited to give more resolution (DR) than the actual converter (sensor pixel).
And that must be why an 8KHz downsampling of a CD has more than double the dynamic range of the original, right? You can google theorems, fine, but apparently have no idea what they mean and where they are applicable. So you must work at DXO, then?

Try a test--take a D800 photo of Stouffer wedges, push exposure in post how much you like, and make a note of the darkest one. Now downsample to 1/64, i.e. ~0.5mpix and repeat. How many extra wedges appeared? Exactly as many as how much the *oversampling theorem* is applicable here--i.e. zero.
 
Upvote 0
straub said:
And that must be why an 8KHz downsampling of a CD has more than double the dynamic range of the original, right? You can google theorems, fine, but apparently have no idea what they mean and where they are applicable. So you must work at DXO, then?
The downsampling will reduce the bandwidth but increase the resolution of the samples. You trade off precision in the amplitude (quantisation noise) for bandwidth (loosing high frequencies). This is exactly the same as happens when you down sample a digital image - and just as you might not want to listen to all your CDs at 8KHz you probably don't want to print out all your images at 600x400 no matter how much this improves the DR.

I posted the Google links as they are usefully informative on the subject. You, rather than posting an informed and referenced response, just posted an insult (FYI, I am a Physicist who has worked with DSP and signal processing for more than 20 years, and I have no connection to DXO).

straub said:
Try a test--take a D800 photo of Stouffer wedges, push exposure in post how much you like, and make a note of the darkest one. Now downsample to 1/64, i.e. ~0.5mpix and repeat. How many extra wedges appeared? Exactly as many as how much the *oversampling theorem* is applicable here--i.e. zero.
Well, I do not have a D800 to test (and have not commented on this specifically).

I find that the DXO measurement data matches pretty well my experience when shooting with those cameras that I have (GF1, 40D, 5DII, 5DII). You can take issue with the fairly arbitrary criteria that they use to produce their overall scores and I am sure that DXO sometimes make mistakes, but the individual measurements - if you understand what they are - have always looked pretty good to me and correlate well with my experience of the 5DIII (particularly when compared to the 5DII).

Simply wanting something to be different does not make it so.
 
Upvote 0
I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Thanks.
 
Upvote 0
Status
Not open for further replies.