July 30, 2014, 04:53:32 PM

Author Topic: DXO vs Reality  (Read 10837 times)

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: DXO vs Reality
« Reply #30 on: March 29, 2012, 12:04:08 PM »
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites.  There are many shades of grey.

DxO has some very helpful tools and software.  Just their sensor tests don't cut the mustard, imho.

Regards

Paul

Paul
since English is not my first language, sometimes it's more difficult for me to grade shades of grey and also sometimes it's  more difficult to find what's thrown into one basket :) Anyway you clearly state what you like and dislike in DxO and this is fair. I just have problems with statements (it's not to you, Paul) I read: "they are wrong because the results are not what I expected, what I invested in, what I like, what I believe....".
It's obviously not logic because of this simple schema:
1. Did they describe their testing procedures? Yes
2. Do they follow their testing procedures during tests? Most probably, they risk too much.
3. Is there anything wrong with their testing procedures? If there is, then I'd like to read about that - why there is something wrong with the way they run their test.
4. Is there anything wrong with the interpretation of their results? Is yes, then I'd like to read about that.

I simply don't understand people, who fight against numbers, numbers having their interpretation. If someone believes  there is so much wrong with DxO, then there is a simple way to proof it.
In my opinion: I do believe that numbers which DxO publishes are real. There might be sometimes something to interpretation but as far as I didn't see clearly pushing one brand over another and interpreting same or comparative results the opposite way depending on situation, then I'm not convinced they lie.

Paul, do you know what I mean?
Regards, Marek
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

canon rumors FORUM

Re: DXO vs Reality
« Reply #30 on: March 29, 2012, 12:04:08 PM »

Actionpix

  • Power Shot G16
  • **
  • Posts: 43
    • View Profile
Re: DXO vs Reality
« Reply #31 on: March 29, 2012, 12:37:56 PM »
I always am surprised how people can get excited over "theory". "Well diffraction is only theory but no real life." "Well calculating DoF/hyper focal focusing is no real life situation." It is. "Theory", a word "artists" hate, explains why your images get unnecessary soft when you use to much aperture. Theory explains why you do not get the max (technical) out of your images. The same one can say concerning lab tests. They are "no real life situation" but will show the potential of equipment, whether you use it in real life or not. The potential is there and it has values you can compare. If you do not like the result you can argue against "theory" and "lab test" but they are as real as real life. They only are emotionless.

dichiaras

  • Guest
Re: DXO vs Reality
« Reply #32 on: March 29, 2012, 01:02:45 PM »
Why are so many people so much against science and the scientific method?
DxO gives results for a standardized 8mp image as well as a 100% crop. The more important one is the former, since you usually crop just a little your image, and then downsize it. I looked also at the results for DR from dpreview (for 100% crop), and they more or less agree with those from DxO. So at the end of the day DxO and dpreview both agree that Nikon and Sony have better DR than Canon (crop sensors at least). Quoting from the dpreview of the 600D about DR: "Overall this is exactly what we're used to seeing from Canon DSLRs, meaning a little less highlight range by default than is typically obtained from Nikon and Sony competitors ".

DxO rates the image quality given by Canon lenses as higher than that of Nikon lenses: shall we believe that DxO gives unfair advantage to Canon for the lenses and to Nikon/Sony for the sensors? No, that's silly!

Since I'm at it, I think that dpreview is definitely more lenient with Canon than with any other brand: while describing the highlight priority option they write "Turn this on and the 600D captures an extra stop in the highlights, resulting in an overall range that at least matches that of Sony and Nikon models." That's inaccurate, since the Nikon D5100 has a similar option that pushes DR much higher than the 600D's, as written in the D5100 review. Between DxO and dpreview, I think it's the latter to be unfair.

But again, for many people Canon is a religion: they just have faith, and definitely not in numbers. ;)

PS I have a Canon 1100D with four Canon lenses, and am very happy with the IQ that I get for the buck, but before upgrading (which I plan to do as soon as the 650D and 70D come out) I will take a long and accurate look at the DxO test results.
« Last Edit: March 29, 2012, 01:11:33 PM by dichiaras »

marekjoz

  • 1D Mark IV
  • ******
  • Posts: 945
    • View Profile
    • marekjoz @flickr
Re: DXO vs Reality
« Reply #33 on: March 29, 2012, 01:11:38 PM »
I always am surprised how people can get excited over "theory". "Well diffraction is only theory but no real life." "Well calculating DoF/hyper focal focusing is no real life situation." It is. "Theory", a word "artists" hate, explains why your images get unnecessary soft when you use to much aperture. Theory explains why you do not get the max (technical) out of your images. The same one can say concerning lab tests. They are "no real life situation" but will show the potential of equipment, whether you use it in real life or not. The potential is there and it has values you can compare. If you do not like the result you can argue against "theory" and "lab test" but they are as real as real life. They only are emotionless.

Sorry but it's Throwing tests away is like saying: "I don't care how many HP has my engine. I don't care how fast does it accelerate. Those are just numbers." o
Of course - those are just numbers comparing if one car is faster than the other. That's about numbers as people like to see things comparable. It doesn't say what comfort is inside, nor where and how gently you drive.
Another example - I don't care if my camera/lens resolves 1000lp/mm or 400 = I don't care if my camera has 8MP or 20MP = I don't care if my camera shoots 8fps or 3fps, 14 bits TR or 8 bits TR and so on...
Let's not get crazy. Not everything is comparable but there are important things that are. Better camera will not make a photographer of anybody - that's obvious.  But why not look on comparisons, tests or numbers? I don't get it.
Charts and graphs are simple enhancement to specification. What's wrong in it?
« Last Edit: March 29, 2012, 01:16:18 PM by marekjoz »
flickr | youtube | 5D2, 50 F/1.4, 24-105 F/4 L IS, 300 F/4 L IS, x1.4 II

awinphoto

  • 1D X
  • *******
  • Posts: 1968
    • View Profile
    • AW Photography
Re: DXO vs Reality
« Reply #34 on: March 29, 2012, 01:15:00 PM »
For the most part, my view on DxO is similar to how i take DPR and the rest of the "in-depth" review sites... with a grain of salt.  Getting a top of the line camera isn't going to make you a better photographer... It CAN however give you a higher ceiling to learn... It CAN give you more options and make things easier, faster, cleaner, but it will vary from photographer to photographer.  Nikon, in theory, has typically scored higher than canon in the last half decade in these sort of tests, however a good chunk of National Geo's staff and field photographers shoot canon and get great images. 

As far as DR and such, as another has posted, in post, it gives more flexibility and room for editing... while that's fine and good, your forgetting a lot of "pro" photographers, when it comes to photoshop or post production, are lazy.  You would be surprised how many pro's outsource photoshop work.  You would also be surprised how many dont outsource but  "shoot to print"... They use an expodisc or the like, nail WB, nail exposure, shoot, and then do little in post to prep the files... If all of a sudden the files are turning up flat, that isn't going to sit well as it defeats their business motto.  Of course there are some "low volume" photographers who charge more per shoot, they can afford spending time in PS or LR and clean up images all day long, but that is going to get old quick.  So there's different angles to look at everything. 

Lastly, coming from a background of shooting cameras that weren't the most opportunistic cameras, such as toyo's for LF instead of sinars, bronicas and kowas instead of Mayima and such... I have learned to adapt, to make the best of my equipment, and not to worry so much about numbers/test/reports as if i have, I would have dropped out a long time ago...  In the end, it's not the equipment nor what some test says about the equipment, it's about how you use said equipment. 
Canon 5d III, Canon 24-105L, Canon 17-40L, Canon 70-200 F4L, Canon 100L 2.8, 430EX 2's and a lot of bumps along the road to get to where I am.

awinphoto

  • 1D X
  • *******
  • Posts: 1968
    • View Profile
    • AW Photography
Re: DXO vs Reality
« Reply #35 on: March 29, 2012, 01:29:21 PM »
Why are so many people so much against science and the scientific method?
DxO gives results for a standardized 8mp image as well as a 100% crop. The more important one is the former, since you usually crop just a little your image, and then downsize it. I looked also at the results for DR from dpreview (for 100% crop), and they more or less agree with those from DxO. So at the end of the day DxO and dpreview both agree that Nikon and Sony have better DR than Canon (crop sensors at least). Quoting from the dpreview of the 600D about DR: "Overall this is exactly what we're used to seeing from Canon DSLRs, meaning a little less highlight range by default than is typically obtained from Nikon and Sony competitors ".

DxO rates the image quality given by Canon lenses as higher than that of Nikon lenses: shall we believe that DxO gives unfair advantage to Canon for the lenses and to Nikon/Sony for the sensors? No, that's silly!

Since I'm at it, I think that dpreview is definitely more lenient with Canon than with any other brand: while describing the highlight priority option they write "Turn this on and the 600D captures an extra stop in the highlights, resulting in an overall range that at least matches that of Sony and Nikon models." That's inaccurate, since the Nikon D5100 has a similar option that pushes DR much higher than the 600D's, as written in the D5100 review. Between DxO and dpreview, I think it's the latter to be unfair.

But again, for many people Canon is a religion: they just have faith, and definitely not in numbers. ;)

PS I have a Canon 1100D with four Canon lenses, and am very happy with the IQ that I get for the buck, but before upgrading (which I plan to do as soon as the 650D and 70D come out) I will take a long and accurate look at the DxO test results.

It's not so much that we are against scientific tests, it's just not the all to be all... in this day and age, we are so competitive, so eager to be #1 that when something isn't #1, we get panicky... Case in point, D800 scores well in the DxO scores....  you saw how the forum threads blew up... oh what has canon done, nikon is so much better, DR this, DR that, when will canon catch up... shoot that basically summarized two full days on Canon Rumors last week.  Even IF, and this is a big IF, canon comes even within 2-3 points on the ratings, how does that fair out when pairing Canons lenses and nikons lenses... All the obsessing over sensors is like the film days... which film has better DR, which film has the better color/saturation, which has more sensitivity, etc... The difference is we cant interchange them like we could with film... There are so many more factors now when evaluating cameras as it is a whole package... AF, build, weather sealing, on-board software, usability, etc... it really is a complete package rather than a single element, and this test is only testing one single element.  Lastly DPR, in going back through their paces, in comparisons with nikons offerings... the 7D and the initial 5D were the first cameras they gave glowing reviews of... more or less they picked apart the xxd series to death, picked apart the 5d2 compared to the D700, and only of recent have they really warmed up to Canon, within the last 2-3 years.  At least their tests are more of the camera as a whole, which I can give more weight in as a "review" site. 
Canon 5d III, Canon 24-105L, Canon 17-40L, Canon 70-200 F4L, Canon 100L 2.8, 430EX 2's and a lot of bumps along the road to get to where I am.

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DXO vs Reality
« Reply #36 on: March 30, 2012, 07:14:53 PM »


Your making my point.  Sorry.  The sensor in the d800 is not leaps and bounds over that of the D4.

I know that.
Nikon knows that.
DxO tests can't figure that out.

DxO scores of the D4 and D800 are pretty close -- basically neck and neck except for the dynamic range part of the test, hardly different by "leaps and bounds". The D4 actually does slightly better on the ISO score (2965 vs 2853)

The only substantial difference on the test is that the D800 has more dynamic range (about a stop more according to the test) at ISO100 and ISO200.

canon rumors FORUM

Re: DXO vs Reality
« Reply #36 on: March 30, 2012, 07:14:53 PM »

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3748
  • POTATO
    • View Profile
    • Nature Photography
Re: DXO vs Reality
« Reply #37 on: March 30, 2012, 07:38:40 PM »
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D

I take DXO results with a nice, big, honkin grain of salt most of the time. I like the ability to include consistently-generated (hmm, grain of salt there?) low-level hardware statistics as a factor in comparing like-brand cameras (i.e. I am looking forward to seeing if they measure any hardware-level improvements between the 5D II and III), as most of the time I figure that stuff should be pretty accurate. However when it comes to some of their numbers and some of their customers (i.e. Nikon), I become more and more skeptical. With their claim of 14.4 stops of "print" DR for the Nikon D800, some 2/3rds of a stop better than the sensor itself is capable of, I've become extremely skeptical of their numbers, and usually try to have a few big, honkin grains of salt handy at all times.

When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14. DXO's DR ratings are only useful if you are shooting a scene with extreme DR to start with, and you push the "expose to the right" (ETTR) technique to the absolute limits. Even then, human error and caution will prevent you from actually achieving the maximum possible DR as indicated by DXO. The only way to actually use all of a camera's available DR is to expose a scene you simply can't expose properly no matter how much headroom it has, in which case you'll always end up with unrecoverable blocked blacks and blown highlights.
My Photography
Current Gear: Canon 5D III | Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: SBIG STT-8300M | Canon EF 300mm f/2.8 L II

iMagic

  • Rebel SL1
  • ***
  • Posts: 99
    • View Profile
Re: DXO vs Reality
« Reply #38 on: March 30, 2012, 07:39:14 PM »
I could care less about technical this and that. Bottom line, what are real world results preceptually (both on screen and in print) and what are the real world practicalities (useability, accessories, etc.). For example, if the final result is a print, does the DR and mega pixels really make a difference on the size/quality of print?

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3304
    • View Profile
Re: DXO vs Reality
« Reply #39 on: March 31, 2012, 12:04:55 AM »
I'm just wondering, alot of people get REALLY worked up over these dxo tests, however theire numbers relating to various cameras (to me anyway) dont appear to reflect real world results take the medium format digital backs for example, these are simply amazing yet score lower than a sony or a nikon?

Personally i dont put any faith in this sort of analysis :P

Note: lucky i cant get smited to death by the DxO brigade :D

I take DXO results with a nice, big, honkin grain of salt most of the time. I like the ability to include consistently-generated (hmm, grain of salt there?) low-level hardware statistics as a factor in comparing like-brand cameras (i.e. I am looking forward to seeing if they measure any hardware-level improvements between the 5D II and III), as most of the time I figure that stuff should be pretty accurate. However when it comes to some of their numbers and some of their customers (i.e. Nikon), I become more and more skeptical. With their claim of 14.4 stops of "print" DR for the Nikon D800, some 2/3rds of a stop better than the sensor itself is capable of, I've become extremely skeptical of their numbers, and usually try to have a few big, honkin grains of salt handy at all times.

Not what you said before they posted D800 results....

Quote
When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14. DXO's DR ratings are only useful if you are shooting a scene with extreme DR to start with, and you push the "expose to the right" (ETTR) technique to the absolute limits. Even then, human error and caution will prevent you from actually achieving the maximum possible DR as indicated by DXO. The only way to actually use all of a camera's available DR is to expose a scene you simply can't expose properly no matter how much headroom it has, in which case you'll always end up with unrecoverable blocked blacks and blown highlights.

Who cares what those details are the point is to use it to compare cameras relative to one another.

takoman46

  • Guest
Re: DXO vs Reality
« Reply #40 on: March 31, 2012, 01:17:27 AM »
I agree that the DxO analysis does not prove that any camera is definitively superior to another camera when it comes to real world applications.  It surprises me how many people gawk at the ratings and results of the DxO tests and base their purchasing decisions on them.  For example, based on DxO results, the logical decision would be to purchase a D800 over a D4 or D3s even if cost was not a factor; simply because DxO tests "prove" that the D800 is superior to a D4 or D3s. Now think about it... The D4 and D800 were released in the same quarter so Nikon obviously had the opportunity to choose how they wanted the D4 and D800 sensors to be built.  So Nikon puts their "greatest sensor ever" into the D800 which is not their flagship model. To me a "flagship model" defines what a company is made of and is the pinnacle of a company's achievements in research and development.  So according to DxO... Nikon somehow chose not to put their "best sensor" in the D4... right... that's completely illogical and I'm sure also not true... yes, the D800 has 36 megapixels... but will it outperform a D4 in all other areas that contribute to producing a great image? I think we should ask someone who owns both a D4 and D800 to see how they feel after working with both cameras.

The same would go for Canon with the 5DmkIII and 1Dx. Although, nobody is saying that the 5DmkIII is better than a 1Dx; the 5DmkIII has more megapixels than a 1Dx. Maybe not as dramatic a jump in comparison to what's going on in the Nikon world, but the general difference still remains. Does anyone think that Canon put their "best sensor" in the 5DmkIII and put a crappier sensor in the 1Dx? What if DxO said so? Would you believe it?

Just to add support for my reasoning... (BTW, this is a non-photography related example). What if Motor Trend magazine did a test between the 2012 Chevrolet Camaro ZL1 and 2012 Corvette ZR-1 and said that the Camaro was the best sports coupe that Chevrolet ever came out with? Kind of a no brainer right? LOL
« Last Edit: March 31, 2012, 01:25:05 AM by takoman46 »

briansquibb

  • Guest
Re: DXO vs Reality
« Reply #41 on: March 31, 2012, 01:41:35 AM »
When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14.

The histogram in DPP seems to show DR - is that correct and accurate?

I aim for maximum DR of the shots I am taking with the 1Ds3 and max out (DPP reading) at around 10, getting mostly over 9 for iso 100/200

The equivalent for the 1D4 and 7D are about 9 max with average about 8

These are for shots that should have high DR

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3748
  • POTATO
    • View Profile
    • Nature Photography
Re: DXO vs Reality
« Reply #42 on: March 31, 2012, 01:55:21 AM »
When it comes to DR, if you "expose correctly" in-camera for the scene you'll never actually use 12 stops, let alone 14.

The histogram in DPP seems to show DR - is that correct and accurate?

I aim for maximum DR of the shots I am taking with the 1Ds3 and max out (DPP reading) at around 10, getting mostly over 9 for iso 100/200

The equivalent for the 1D4 and 7D are about 9 max with average about 8

These are for shots that should have high DR

Those sounds like entirely reasonable numbers to me...
My Photography
Current Gear: Canon 5D III | Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: SBIG STT-8300M | Canon EF 300mm f/2.8 L II

canon rumors FORUM

Re: DXO vs Reality
« Reply #42 on: March 31, 2012, 01:55:21 AM »

pj1974

  • 7D
  • *****
  • Posts: 373
    • View Profile
    • A selection of my photos (copyright)
Re: DXO vs Reality
« Reply #43 on: March 31, 2012, 03:19:15 AM »
I've also found DxO's camera sensor tests to be quite meaningless, when relating to the real world. They don't weight the criteria used in their tests well, imho. Often sensors which are given 'high' DxO ratings, don't perform as well across a number of 'real life situations' as other sensors which are given lower DxO ratings.

There are many other websites that cite themselves as professional, systematic, etc - whereas there are SO many variables, and if they don't get 1 thing just 'spot on [......................]
Cheers.

Paul

Well stated but honestly it's not fair putting in one row DxO and such sites you've mentioned...

Hi marekjoz

Thanks for your comment. I wasn't at my usual computer when I wrote my previous post, so I couldn't look up the sites I was referring to.

It would probably have been more helpful to explain that I'm not placing DxO in the same basket as many other even less systematic photographic equipment testing sites.  There are many shades of grey.

DxO has some very helpful tools and software.  Just their sensor tests don't cut the mustard, imho.

Regards

Paul

Paul
since English is not my first language, sometimes it's more difficult for me to grade shades of grey and also sometimes it's  more difficult to find what's thrown into one basket :) Anyway you clearly state what you like and dislike in DxO and this is fair. I just have problems with statements (it's not to you, Paul) I read: "they are wrong because the results are not what I expected, what I invested in, what I like, what I believe....".
It's obviously not logic because of this simple schema:
1. Did they describe their testing procedures? Yes
2. Do they follow their testing procedures during tests? Most probably, they risk too much.
3. Is there anything wrong with their testing procedures? If there is, then I'd like to read about that - why there is something wrong with the way they run their test.
4. Is there anything wrong with the interpretation of their results? Is yes, then I'd like to read about that.

I simply don't understand people, who fight against numbers, numbers having their interpretation. If someone believes  there is so much wrong with DxO, then there is a simple way to proof it.
In my opinion: I do believe that numbers which DxO publishes are real. There might be sometimes something to interpretation but as far as I didn't see clearly pushing one brand over another and interpreting same or comparative results the opposite way depending on situation, then I'm not convinced they lie.

Paul, do you know what I mean?
Regards, Marek

Hi Marek

Yes, I do know what you mean.  It's true that a lot of 'testing websites' meet the conditions you write about.  What annoys me, is that the 'ratings' that these websites give do not always take into account many other factors (even in terms of pure image quality / sensor capabilities) - and 'not including that' - is a 'gap' - that should does not represent professionalism.

So I don't really place any of my purchasing decision on DxO's sensor tests. But they do have some useful software I'm VERY glad to have bought!

By the way, congratulations with good use of English!  I speak 5 languages, so I know what it's like trying to express thoughts in a language that is not one's mother language.

Regards

Paul
I'm not a brand-fanatic. What I do appreciate is using my 7D and 350D cameras along with a host of lenses & many accessories to capture quality photos, and share with friends.

maxxevv

  • Guest
Re: DXO vs Reality
« Reply #44 on: March 31, 2012, 08:28:47 AM »

Sorry but it's Throwing tests away is like saying: "I don't care how many HP has my engine. I don't care how fast does it accelerate. Those are just numbers." o
Of course - those are just numbers comparing if one car is faster than the other. That's about numbers as people like to see things comparable. It doesn't say what comfort is inside, nor where and how gently you drive.
Another example - I don't care if my camera/lens resolves 1000lp/mm or 400 = I don't care if my camera has 8MP or 20MP = I don't care if my camera shoots 8fps or 3fps, 14 bits TR or 8 bits TR and so on...
Let's not get crazy. Not everything is comparable but there are important things that are. Better camera will not make a photographer of anybody - that's obvious.  But why not look on comparisons, tests or numbers? I don't get it.
Charts and graphs are simple enhancement to specification. What's wrong in it?

Well, there is some reality in that but also some error in the parallels.

In the context of the car, the DXO results are often more like "volume of air exhausted", "heat dissipation rate" and "150-200km/h rate".

In the grand scheme of things in practically using a car or camera, they are mostly moot ... 

canon rumors FORUM

Re: DXO vs Reality
« Reply #44 on: March 31, 2012, 08:28:47 AM »