April 23, 2014, 11:10:06 AM

Author Topic: DXO vs Reality  (Read 10098 times)

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: DXO vs Reality
« Reply #15 on: March 28, 2012, 11:53:15 PM »
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on' - it can ruin the overall results.  I recall one site that tested scores of lenses, but many of them at their very minimum focusing distance (MFD) - where some lenses (even high quality ones) are not at their sharpest... and the results were very skewed.

Another site I came across a few weeks ago - the testers only tested the 'minimum focal length' (eg 70-300mm @ 70mm - and all at f8) - and only a certain 'part' of the overall image (centre pixels).  It was crazy how some average quality lenses were ranked the same as others which were much higher, just based on that.

I have a marketing management and accounting degree, so I understand about corporate logos, branding, sponsorship, 'marketing words', etc - very well.  So that could be an issue... but I doubt that DxO would test eg Nikon more favourably just because of that (they could open themselves up to legal action if that were the case....)

Having said that, I do like DxO's Optics Pro software a LOT.  So I'm a customer of DxO's.  I won't be so 'put off' by the annoyances I have with DxO's sensor test not to use their good stuff!

Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

canon rumors FORUM

Re: DXO vs Reality
« Reply #15 on: March 28, 2012, 11:53:15 PM »

Aglet

  • 1D Mark IV
  • ******
  • Posts: 814
    • View Profile
    • A2B Art
Re: DXO vs Reality
« Reply #16 on: March 29, 2012, 01:21:22 AM »
Poking my head around the Canon side cause I have pretty much no commitment to Nikon as far as lenses are considered (hell I use a Canon EOS strap on my D5100...)

Funny.  I just bought a D5100 to see if it can really perform better in real life High-DR shots better than my semi-pro Canon gear. :)

preliminary results are looking like it can... More testing to do but making some public soon.

Wrathwilde

  • Rebel T5i
  • ****
  • Posts: 139
    • View Profile
    • Anarchy Photography
Re: DXO vs Reality
« Reply #17 on: March 29, 2012, 02:32:36 AM »
I use a Canon EOS strap on my D5100.

Camera straps are often overlooked, it's a crucial piece of kit that can make or break your camera. Ken Rockwell once compared camera straps from all the major manufactures, and the ones that didn't make their own straps he was able to simulate by taking a generic strap and writing the manufactures name on it in nail polish that most closely mimicked the color of the mfg's logo. The results were astounding!!! With the Zeiss strap attached the images became much sharper and popped with an almost 3D quality to them. The Leica strap increased percievable resolution by 20%. The Nikon strap increased the DR but just didn't feel right, a bit clunky and cumbersome. The Canon EOS strap was the best all around, there was a Hit to the DR, but was more comfortable than any of the other straps and was right behind the Leica and Zeiss straps when it came to overall image quality.

Cheers,
Wrathwilde
« Last Edit: March 29, 2012, 02:36:43 AM by Wrathwilde »

Aglet

  • 1D Mark IV
  • ******
  • Posts: 814
    • View Profile
    • A2B Art
Re: DXO vs Reality
« Reply #18 on: March 29, 2012, 02:41:06 AM »
My take on DxOmark.

Take w a spoon of pepto but I now prefer the SCREEN comparison on DxO rather than the PRINT version.

I print big, their test print is 12x18 inches i think, at about 180 or 200ppi to = 8MP. 

18x12 is the smallest print size I make, which matches up well to the 10MP output of my old 40D, for example.

If you look at the screen comparisons for a lot of cameras what you'll find is that this pixel-level type of result is very good at showing you how various camera systems can perform when you're viewing on screen at 1:1 size.  Go ahead, check your favorite cameras... I'll wait...

Done?

OK.  notice how much more similar SCREEN results on most of them are at:

- 18% SNR?
- Tonal Range?
- Color Sensitivity?
- Dynamic Range?  ( more on this one later )

What's that telling us?
To me it looks like most camera system mfrs are doing a pretty good job pressing the limits of physics and electronics.

If you look at cameras that are at or below about 10MP, SCREEN and PRINT numbers are closer than they are for higher rez cameras of a similar type.  It's once you get above about 10 MP sensors that the S vs P numbers start to show more of a difference, with PRINT pulling ahead.
Why?
Likely because the merging of pixel data from higher resolution sensors into few effective pixels for print tend to average out noise and other pixel-level inconsistencies.
I think if the test print was 30x20 inches at 200ppi we'd see different results until sensor resolutions exceed 24MP.

SO, if you're using more than 8MP worth of sensor data to print a similar 18x12" print at 200ppi you're likely getting dimishing returns from higher resolution gear but your results may align better to the PRINT results and actually look a little better.. however it is that they actually measure that.

If you're printing huge sizes where individual pixel data starts to become visible at nose-to-paper distances then your results may be more closely related to the SCREEN results.

Back to the Dynamic Range spec.  This is where Canon's gear is reaching its limits at iso settings of 800 and lower compared to the competition.  At the limited print size used for the DxO test this is still not likely to show very much of the low iso banding problems some of us hardware-pushing types complain about.  If you only print little postcards sizes like 8x10 or 11x14 ( ;) ) you're not very likely to notice banding except perhaps on screen at 1:1 size.

I've used a 7D that showed significant low-iso banding that's sometimes so bad it's very visible on screen with only a slight tone-curve tweak to bring up some shadow area details.  There are vertical bands 8 pixels wide, 8 pixels apart, across shadow areas of low iso images.  Even if making an 18x12" print that's at 300ppi, that's less than 20 line-pairs of noise per inch, something that can quite easily be noticed.

I also shoot with a 60D, the DxO scores for which are virtually identical to the 7D with the latter tending to rate slightly higher.
Funny thing about the results tho, the 60D suffers considerably less low-iso banding issues than the 7D.  They're not showing up in the shadows of my larger prints or on screen, certainly not to anywhere near the extent they are from the 7D.  If I were to look at the DxO results, the 7D looks better by a hair.  Real life is not the case in my experience.

I don't know exactly what criteria or methods DxO is using for their SNR and DR measurements, whether SCREEN or PRINT. I'm not sure if they take into account this sort of pattern noise that some cameras add too much of to an image.  If averaging out the noise data, the banding will have less of an effect on the measured specs yet still show up as a problem in images. 
Still, the better the SNR and DR numbers are then the less likely this is to be a problem. 

I'm looking forward to seeing how much of DxO's reported 2.5 stops of base-iso DR lead for the D5100 translate into real-world usability over a 60D or 7D.  And maybe over the 5D2 as well.
Would that make the $650 marvel a better camera than the 5D2?  heck no!  but it makes it worth using in situations I now know some of my Canon gear will not be able to handle as well.

Just trying to use the best tool for the job.

Ivar

  • Rebel T5i
  • ****
  • Posts: 143
    • View Profile
Re: DXO vs Reality
« Reply #19 on: March 29, 2012, 04:37:29 AM »
Reality seems to confirm better DR of some cameras.

The question is what you read out of the total scores, it may help quite well with your decision, there is no other such a methodological approach as far as I know.

Mind you they measure only sensors, irrespective of content and in that sense they describe sensor basic capabilities.

Clear mind is still needed.


I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D

infared

  • 5D Mark III
  • ******
  • Posts: 745
  • Kodak Brownie!
    • View Profile
Re: DXO vs Reality
« Reply #20 on: March 29, 2012, 05:51:47 AM »
I'm a Nikon shooter currently with a D5100. DxOMark ranks the D5100/D7000 very well in terms of DR, but I can never really get the amount of dynamic range that I'm happy with in my photos. This is primarily why I'm really looking to go full frame.

Just goes to show you how useless numbers are.

The DR ratings are like gas milage ratings.  They are the best ideal case, but not what most users see.  The DR ratings for jpeg are closer to reality.  You can get more DR from Raw files, but the images more often than not look flat and awful.

Raw files are neutral..and FULL of information. That is the beauty of them. Yes they look flat. Take them to post and you have SO MUCH MORE information to pull out the image to create what you saw when you took the photo. Yes it takes more time, yes you need good software, (more than one in my opinion, depending on what the final image will look like)..but Raw Files are an incredible tool.  They are the whole reason why I spend my hard earned money on a expensive camera   that produces them.  I know a guy who owns an  5D Mark II and uses it to shoot weddings in jpeg?  Me thinks he might get a LITTLE more detail out of the bride's dress if he shot in Raw but my suggestions fell on deaf ears?
5D Mark III, Canon 15mm f/2.8 Fisheye, Canon 17mm f/4L TS-E, Canon 16-35mm f/2.8L II, 21mm f/2.8 Zeiss, Sigma 35mm f/1.4, 24-70mm f/2.8 II, 50mm f/1.4 Sigma, 85mm f/1.2L, 100mm f/2.8L Macro,70-200mm f/2.8L IS II...1.4x converter III, and some other stuff.....

dilbert

  • 1D X
  • *******
  • Posts: 2396
    • View Profile
Re: DXO vs Reality
« Reply #21 on: March 29, 2012, 05:56:24 AM »
I never heard or read about any complaints relating to DxO on this website until the D800 was tested.

Similarly, if the 1DX beats the D4 at DxO, I'm sure lots of people here how good the 1DX is based on DxO (and they'll point to photos here and there that show it.)

Ho hum.

I just wish people would be honest about why they want to dismiss, diminish and rubbish DxO test results.

canon rumors FORUM

Re: DXO vs Reality
« Reply #21 on: March 29, 2012, 05:56:24 AM »

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DXO vs Reality
« Reply #22 on: March 29, 2012, 06:31:55 AM »
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

You have to look beyond the one number summary and look at the graphs/scores. The medium format backs don't score as high because they don't good high ISO numbers. I don't know these products well, but it looks like they're not designed for high ISO shooting (Some simply won't let you push the ISO very high).

I suspect the real reason DxO is coming under fire all of a sudden has more to do with the Nikon D800's score than the scores of Medium format cameras.

well_dunno

  • 7D
  • *****
  • Posts: 356
    • View Profile
Re: DXO vs Reality
« Reply #23 on: March 29, 2012, 07:33:18 AM »
I did mention a few times prior to D800 testing that DxO suggested Canon 70-200 f/2.8 IS mark 1 was sharper than mark 2. Even after forum complaints there  that the results could not be correct, DxO maintained that there was nothing wrong with the results.

http://www.dxomark.com/index.php/News/DxOMark-news/Canon-EF-70-200mm-f-2.8L-IS-II-USM-measurements-and-review

This is probably the only source that suggests mark 1 was sharper than mark 2. I am not suggesting DxO is necessarily biased, but something does not seem right. Perhaps their testing just does not have enough coverage to give a realistic picture?

Either case, I look at their figures but never rely on them for any purchase...

Cheers!
« Last Edit: March 29, 2012, 07:35:25 AM by well_dunno »

Maui5150

  • 7D
  • *****
  • Posts: 336
    • View Profile
Re: DXO vs Reality
« Reply #24 on: March 29, 2012, 07:52:09 AM »
I did mention a few times prior to D800 testing that DxO suggested Canon 70-200 f/2.8 IS mark 1 was sharper than mark 2. Even after forum complaints there  that the results could not be correct, DxO maintained that there was nothing wrong with the results.

http://www.dxomark.com/index.php/News/DxOMark-news/Canon-EF-70-200mm-f-2.8L-IS-II-USM-measurements-and-review

This is probably the only source that suggests mark 1 was sharper than mark 2. I am not suggesting DxO is necessarily biased, but something does not seem right. Perhaps their testing just does not have enough coverage to give a realistic picture?

Either case, I look at their figures but never rely on them for any purchase...

Cheers!


A couple thoughts come to mind...

For me it was not so much the Nikon D800 score, but it was that score relative to other cameras I know to be better and more capable.  If you go by DxO, the D4 and D3s are shite compared to the D800 which is just not true.  Then when you throw the whole MF quality into the mix... Yes, the MFs may bot be designed for High ISO per se, but does beg to question the ability to quantitatively measure IQ

In someways reminds me of CDs versus LPs, and while "Digital" is supposed to capture and be so reproduceable, the audiophiles still find magic in the outdated LPs for sound quality. 

Not to bring film versus digital back in the mix, but I do think that it is possible to produce a camera that "tests" well but whose IQ to the eye does not meet with the same judgement. 

To me this diminishes the value of DxO since they are telling me a new BMW outperforms a new Ferrari

Is the D800 a better camera than the D4?  DxO says it is by a big margin. 

You can tell me Rosie O'donnell is hotter than Olivia Wilde, I'll say you just have a thing for fat chicks who aren't funny

pj1974

  • 7D
  • *****
  • Posts: 353
    • View Profile
    • A selection of my photos (copyright)
Re: DXO vs Reality
« Reply #25 on: March 29, 2012, 08:01:04 AM »
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites.  There are many shades of grey.

DxO has some very helpful tools and software.  Just their sensor tests don't cut the mustard, imho.

Regards

Paul
I'm not a brand-fanatic. What I do appreciate is using my 7D and 350D cameras along with a host of lenses & many accessories to capture quality photos, and share with friends.

dilbert

  • 1D X
  • *******
  • Posts: 2396
    • View Profile
Re: DXO vs Reality
« Reply #26 on: March 29, 2012, 08:15:15 AM »
...
For me it was not so much the Nikon D800 score, but it was that score relative to other cameras I know to be better and more capable.  If you go by DxO, the D4 and D3s are S___e compared to the D800 which is just not true.  Then when you throw the whole MF quality into the mix... Yes, the MFs may bot be designed for High ISO per se, but does beg to question the ability to quantitatively measure IQ

Then don't look at the score, look at the signal to noise ratio, colour depth, tonality and dynamic range graphs individually to see how the D800 scores relative to those other cameras. The thing is, DxO didn't give the D800 for one attribute alone, it was from many.

If you compare the DR of the D800 to any other DSLR, you'll find that at ISO under 400, and especially ISO 100, it has no competition.

Quote
To me this diminishes the value of DxO since they are telling me a new BMW outperforms a new Ferrari

That's not unheard of.

Just because a new Ferrari is new does not mean it is as good as or better than any other car.

Quote
Is the D800 a better camera than the D4?  DxO says it is by a big margin.

DxO doesn't measure/score "camera", it scores "sensor." Thus DxO ranks camera based on sensor and sensor alone. For some people that will make it a better camera than the D4, for some it will not.

Maui5150

  • 7D
  • *****
  • Posts: 336
    • View Profile
Re: DXO vs Reality
« Reply #27 on: March 29, 2012, 08:46:19 AM »
...
For me it was not so much the Nikon D800 score, but it was that score relative to other cameras I know to be better and more capable.  If you go by DxO, the D4 and D3s are S___e compared to the D800 which is just not true.  Then when you throw the whole MF quality into the mix... Yes, the MFs may bot be designed for High ISO per se, but does beg to question the ability to quantitatively measure IQ

Then don't look at the score, look at the signal to noise ratio, colour depth, tonality and dynamic range graphs individually to see how the D800 scores relative to those other cameras. The thing is, DxO didn't give the D800 for one attribute alone, it was from many.

If you compare the DR of the D800 to any other DSLR, you'll find that at ISO under 400, and especially ISO 100, it has no competition.

Quote
To me this diminishes the value of DxO since they are telling me a new BMW outperforms a new Ferrari

That's not unheard of.

Just because a new Ferrari is new does not mean it is as good as or better than any other car.

Quote
Is the D800 a better camera than the D4?  DxO says it is by a big margin.

DxO doesn't measure/score "camera", it scores "sensor." Thus DxO ranks camera based on sensor and sensor alone. For some people that will make it a better camera than the D4, for some it will not.

Your making my point.  Sorry.  The sensor in the d800 is not leaps and bounds over that of the D4.

I know that.
Nikon knows that.
DxO tests can't figure that out.

Despite what DxO is trying to tell me, Rosie O'Donnell is still fat and not more attractive than Olivia Wilde. 

Fact.  Not Opinion. 

canon rumors FORUM

Re: DXO vs Reality
« Reply #27 on: March 29, 2012, 08:46:19 AM »

rpt

  • 1D X
  • *******
  • Posts: 2092
  • Could not wait for 7D2 so I got the 5D3
    • View Profile
Re: DXO vs Reality
« Reply #28 on: March 29, 2012, 11:00:33 AM »
DxO doesn't measure/score "camera", it scores "sensor." Thus DxO ranks camera based on sensor and sensor alone. For some people that will make it a better camera than the D4, for some it will not.

Gold!

You hit the head one the nail (or the other way around... whatever... )

itsnotmeyouknow

  • EOS M2
  • ****
  • Posts: 238
    • View Profile
Re: DXO vs Reality
« Reply #29 on: March 29, 2012, 11:09:16 AM »
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?


You have to look beyond the one number summary and look at the graphs/scores. The medium format backs don't score as high because they don't good high ISO numbers. I don't know these products well, but it looks like they're not designed for high ISO shooting (Some simply won't let you push the ISO very high).

I suspect the real reason DxO is coming under fire all of a sudden has more to do with the Nikon D800's score than the scores of Medium format cameras.


The only medium format digital camera with an ISO of above 800 is the Pentax 645D which goes up to 1600.  Th ereason being that most, apart from teh Pentax are aimed at studio shooters.  The Pentax, being weather shielded and having a few lenses that are also weather shielded is aimed also at landscape shooters, which is does very well. 

This is a shot I took at a concert in Sweden which was at every extreme:

Pentax 645D A (manual focus) 80 - 160 f/4.5 @ f/4.5 and 160mm 1/125 ISO 1600


hannah with orchestra in sweden by singingsnapper, on Flickr

canon rumors FORUM

Re: DXO vs Reality
« Reply #29 on: March 29, 2012, 11:09:16 AM »