DXO vs Reality

Status
Not open for further replies.
Oct 26, 2011
4,542
1
30,296
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D
 
I received feedback from a client today who wasn't too thrilled with my work. He commented on how the images were poorly lit and composed, and lacked any emotion. Fortunately, I was able to manipulate the exif data to make it appear as if the images were shot with a D800. I sent him a link to its DxOMark test results, explaining how it has the best-performing sensor on earth. He then said, "you're right, these image right here are some medium format $**t. Now everyone's happy ;D
 
Upvote 0
I'm a Nikon shooter currently with a D5100. DxOMark ranks the D5100/D7000 very well in terms of DR, but I can never really get the amount of dynamic range that I'm happy with in my photos. This is primarily why I'm really looking to go full frame.

Just goes to show you how useless numbers are.
 
Upvote 0
Zhanger said:
I'm a Nikon shooter currently with a D5100. DxOMark ranks the D5100/D7000 very well in terms of DR, but I can never really get the amount of dynamic range that I'm happy with in my photos. This is primarily why I'm really looking to go full frame.

Just goes to show you how useless numbers are.

The DR ratings are like gas milage ratings. They are the best ideal case, but not what most users see. The DR ratings for jpeg are closer to reality. You can get more DR from Raw files, but the images more often than not look flat and awful.
 
Upvote 0
Mt Spokane Photography said:
Zhanger said:
I'm a Nikon shooter currently with a D5100. DxOMark ranks the D5100/D7000 very well in terms of DR, but I can never really get the amount of dynamic range that I'm happy with in my photos. This is primarily why I'm really looking to go full frame.

Just goes to show you how useless numbers are.

The DR ratings are like gas milage ratings. They are the best ideal case, but not what most users see. The DR ratings for jpeg are closer to reality. You can get more DR from Raw files, but the images more often than not look flat and awful.

Which is why I really hate processing RAWs, especially on a D5100, which is not exactly a top-notch camera. I find myself just shooting straight JPEGs cause I'm usually pretty good with nailing the exposures the first time around.

Poking my head around the Canon side cause I have pretty much no commitment to Nikon as far as lenses are considered (hell I use a Canon EOS strap on my D5100...)
 
Upvote 0
wickidwombat said:
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D

I agree with you wicki. the more I look at real life sample picture from the various camera like D800, D4 and 5DmkIII at DP Review for example the less I find these DxO test relevant.

If any of you guy are audiophile and sound system lovers, you would know that a cheep sony amplifier often has much better spec sheet then a $10,000 amplifiers! But guess which one sounds better though in real life! (I need to remember this one next time I get paranoid about my camera spec!)

8)
 
Upvote 0
marekjoz said:
Mt Spokane Photography said:
If Nikon were a big customer, and Canon was not, what would you do?

http://www.dxo.com/us/image_quality/customers2

DXO would like you to think they are independent and do not take advertising from camera makers, but ...

Ot the other hand maybe Canon is not present there because doesn't win? ;)

Or doesn't want to "Buy" a high rating?
 
Upvote 0
Definitely agree on DXO discrepancy between lab bench and reality. Wrote a note already on this but the thread was stopped. Imo, this is pure academic exercise. A bit like taking an engine out of a car and test it on a formatted test-bench by white-coat-dressed technicians producing lots of data that nobody gives a damn about. Obviously, a car is much more than an engine. But, the way the data and numbers are presented give the false impression of authenticity. People love that. It's irrational, but they love that. For me, I would not buy any gear just based on DXO reviews. Just take it as informative as any other info available at the time.

Just remember how they found the 70-200/2.8 II to be just OK on their test-bench while absolutely every other review site around the world has only praise for that lens. Rings a bell ?
 
Upvote 0
The thing with testing is that you can test a product to death and find a huge number of issues or come up with fancy charts. However, if your test scenarios do not echo what 80% of your customers would do, there is no point testing. All the time spent in testing, finding issues and fixing them is a waste.
 
Upvote 0
marekjoz said:
Mt Spokane Photography said:
If Nikon were a big customer, and Canon was not, what would you do?

http://www.dxo.com/us/image_quality/customers2

DXO would like you to think they are independent and do not take advertising from camera makers, but ...

Ot the other hand maybe Canon is not present there because doesn't win? ;)

Or Canon has not given DxO permission to put their logo up. In my job, a few of our best customers have a policy of never allowing their logo on such pages. So our "Customer Segments" page does not show their logos.
 
Upvote 0
I will be convinced when 24-70 II will be treated same way.
I know that while testing you can show results in different ways to proof your point the way you want. But they have their testing procedures which are known before each test is performed. If so many people don't believe those tests, then why someone didn't proof yet that they lie?

I'm an engineer and believe in numbers, anyway. I know there are things difficult to measure (like bokeh). But I also know how difficult it's to talk to audiophiles, who pay 1k$ for better USB wire saying it sounds better.

I'm not saying they play fair, but judging that tests of Canon's gear are bad because there's Nikon as customer... Well, I'm not sure....
 
Upvote 0
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on' - it can ruin the overall results. I recall one site that tested scores of lenses, but many of them at their very minimum focusing distance (MFD) - where some lenses (even high quality ones) are not at their sharpest... and the results were very skewed.

Another site I came across a few weeks ago - the testers only tested the 'minimum focal length' (eg 70-300mm @ 70mm - and all at f8) - and only a certain 'part' of the overall image (centre pixels). It was crazy how some average quality lenses were ranked the same as others which were much higher, just based on that.

I have a marketing management and accounting degree, so I understand about corporate logos, branding, sponsorship, 'marketing words', etc - very well. So that could be an issue... but I doubt that DxO would test eg Nikon more favourably just because of that (they could open themselves up to legal action if that were the case....)

Having said that, I do like DxO's Optics Pro software a LOT. So I'm a customer of DxO's. I won't be so 'put off' by the annoyances I have with DxO's sensor test not to use their good stuff!

Cheers.

Paul
 
Upvote 0
pj1974 said:
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on' - it can ruin the overall results. I recall one site that tested scores of lenses, but many of them at their very minimum focusing distance (MFD) - where some lenses (even high quality ones) are not at their sharpest... and the results were very skewed.

Another site I came across a few weeks ago - the testers only tested the 'minimum focal length' (eg 70-300mm @ 70mm - and all at f8) - and only a certain 'part' of the overall image (centre pixels). It was crazy how some average quality lenses were ranked the same as others which were much higher, just based on that.

I have a marketing management and accounting degree, so I understand about corporate logos, branding, sponsorship, 'marketing words', etc - very well. So that could be an issue... but I doubt that DxO would test eg Nikon more favourably just because of that (they could open themselves up to legal action if that were the case....)

Having said that, I do like DxO's Optics Pro software a LOT. So I'm a customer of DxO's. I won't be so 'put off' by the annoyances I have with DxO's sensor test not to use their good stuff!

Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...
 
Upvote 0
Zhanger said:
Poking my head around the Canon side cause I have pretty much no commitment to Nikon as far as lenses are considered (hell I use a Canon EOS strap on my D5100...)

Funny. I just bought a D5100 to see if it can really perform better in real life High-DR shots better than my semi-pro Canon gear. :)

preliminary results are looking like it can... More testing to do but making some public soon.
 
Upvote 0
Zhanger said:
I use a Canon EOS strap on my D5100.

Camera straps are often overlooked, it's a crucial piece of kit that can make or break your camera. Ken Rockwell once compared camera straps from all the major manufactures, and the ones that didn't make their own straps he was able to simulate by taking a generic strap and writing the manufactures name on it in nail polish that most closely mimicked the color of the mfg's logo. The results were astounding!!! With the Zeiss strap attached the images became much sharper and popped with an almost 3D quality to them. The Leica strap increased percievable resolution by 20%. The Nikon strap increased the DR but just didn't feel right, a bit clunky and cumbersome. The Canon EOS strap was the best all around, there was a Hit to the DR, but was more comfortable than any of the other straps and was right behind the Leica and Zeiss straps when it came to overall image quality.

Cheers,
Wrathwilde
 
Upvote 0
My take on DxOmark.

Take w a spoon of pepto but I now prefer the SCREEN comparison on DxO rather than the PRINT version.

I print big, their test print is 12x18 inches i think, at about 180 or 200ppi to = 8MP.

18x12 is the smallest print size I make, which matches up well to the 10MP output of my old 40D, for example.

If you look at the screen comparisons for a lot of cameras what you'll find is that this pixel-level type of result is very good at showing you how various camera systems can perform when you're viewing on screen at 1:1 size. Go ahead, check your favorite cameras... I'll wait...

Done?

OK. notice how much more similar SCREEN results on most of them are at:

- 18% SNR?
- Tonal Range?
- Color Sensitivity?
- Dynamic Range? ( more on this one later )

What's that telling us?
To me it looks like most camera system mfrs are doing a pretty good job pressing the limits of physics and electronics.

If you look at cameras that are at or below about 10MP, SCREEN and PRINT numbers are closer than they are for higher rez cameras of a similar type. It's once you get above about 10 MP sensors that the S vs P numbers start to show more of a difference, with PRINT pulling ahead.
Why?
Likely because the merging of pixel data from higher resolution sensors into few effective pixels for print tend to average out noise and other pixel-level inconsistencies.
I think if the test print was 30x20 inches at 200ppi we'd see different results until sensor resolutions exceed 24MP.

SO, if you're using more than 8MP worth of sensor data to print a similar 18x12" print at 200ppi you're likely getting dimishing returns from higher resolution gear but your results may align better to the PRINT results and actually look a little better.. however it is that they actually measure that.

If you're printing huge sizes where individual pixel data starts to become visible at nose-to-paper distances then your results may be more closely related to the SCREEN results.

Back to the Dynamic Range spec. This is where Canon's gear is reaching its limits at iso settings of 800 and lower compared to the competition. At the limited print size used for the DxO test this is still not likely to show very much of the low iso banding problems some of us hardware-pushing types complain about. If you only print little postcards sizes like 8x10 or 11x14 ( ;) ) you're not very likely to notice banding except perhaps on screen at 1:1 size.

I've used a 7D that showed significant low-iso banding that's sometimes so bad it's very visible on screen with only a slight tone-curve tweak to bring up some shadow area details. There are vertical bands 8 pixels wide, 8 pixels apart, across shadow areas of low iso images. Even if making an 18x12" print that's at 300ppi, that's less than 20 line-pairs of noise per inch, something that can quite easily be noticed.

I also shoot with a 60D, the DxO scores for which are virtually identical to the 7D with the latter tending to rate slightly higher.
Funny thing about the results tho, the 60D suffers considerably less low-iso banding issues than the 7D. They're not showing up in the shadows of my larger prints or on screen, certainly not to anywhere near the extent they are from the 7D. If I were to look at the DxO results, the 7D looks better by a hair. Real life is not the case in my experience.

I don't know exactly what criteria or methods DxO is using for their SNR and DR measurements, whether SCREEN or PRINT. I'm not sure if they take into account this sort of pattern noise that some cameras add too much of to an image. If averaging out the noise data, the banding will have less of an effect on the measured specs yet still show up as a problem in images.
Still, the better the SNR and DR numbers are then the less likely this is to be a problem.

I'm looking forward to seeing how much of DxO's reported 2.5 stops of base-iso DR lead for the D5100 translate into real-world usability over a 60D or 7D. And maybe over the 5D2 as well.
Would that make the $650 marvel a better camera than the 5D2? heck no! but it makes it worth using in situations I now know some of my Canon gear will not be able to handle as well.

Just trying to use the best tool for the job.
 
Upvote 0
Reality seems to confirm better DR of some cameras.

The question is what you read out of the total scores, it may help quite well with your decision, there is no other such a methodological approach as far as I know.

Mind you they measure only sensors, irrespective of content and in that sense they describe sensor basic capabilities.

Clear mind is still needed.


wickidwombat said:
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D
 
Upvote 0
Status
Not open for further replies.