July 30, 2014, 05:00:42 PM

Author Topic: Why the DxO bashing?  (Read 20894 times)

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13616
    • View Profile
Re: Why the DxO bashing?
« Reply #15 on: February 07, 2014, 09:09:03 AM »
Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch. ;)

Many Canon Rumors trolls may experience great satisfaction from demonstrating that ~2 stop deficit in low ISO DR by shooting images with the lens cap on, then pushing those images 4-5 stops in post. 

The majority of Canon devotees, at least here on CR, seem to be fairly objective about the issue.  It's widely acknowledged that Canon sensors deliver less low ISO DR than Nikon/Sony sensors.  The fact remains that people buy cameras, not bare silicon sensors.  If low ISO DR is someone's only criterion for judging a camera's performance (and for a very tiny minority of people, that may be the case), they should choose something other than a Canon camera.  But for most people, what matters is the performance of the system as a whole (camera + lenses + flashes + accessories), and that's where Canon usually wins.   As Don Haines is fond of saying, who cares how many stops of dynamic range a blurry picture has?

Many times in the DR debate, those bashing Canon sensors have been asked to provide examples of shots ruined by Canon's 'poor low ISO DR' that would have been saved by those extra two stops.  Personally, I have almost no examples of that situation - in many scenes, the ~12 stops I get is sufficient, and when the scene DR is greater than 12 stops, it's almost always greater than 14 stops, too.

But for those who would like to persist in this debate, I have found a relevant example showing how an extra two stops of DR can keep the sunlit outdoors from blowing out when shooting an indoor candlelit scene...
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #15 on: February 07, 2014, 09:09:03 AM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 13616
    • View Profile
Re: Why the DxO bashing?
« Reply #16 on: February 07, 2014, 09:10:38 AM »
I cannot help but question the measurement methodology of a group who (either fraudulently or ignorantly) uses biased and misleading summary statistics to put forth claims about camera/lens performance.  That is to say, if you don't analyze your data properly but staunchly claim to be fair and objective, then it is my obligation to question your data collection methods as well, because your entire process is now suspect.  That is what any good scientist does.

DxO has the phrase "Image Science" as part of their logo, but their practices aren't consistent with that phrase.  I'm most concerned by the 'black box' calculation for their summary Sensor Score and Lens Score (methods should be published), and by the fact that they released data which was incorrect, defended it, then subsequently changed it with no acknowledgement of their error.  Also, I'm noticing that the more I delve into their Measurements, the more I find errors (for example, I just looked at the Canon 28-300L measurements and their actuance data shown visually as field maps are ~10% lower relative to the same data plotted on a graph as a profile).
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

AcutancePhotography

  • 6D
  • *****
  • Posts: 468
    • View Profile
Re: Why the DxO bashing?
« Reply #17 on: February 07, 2014, 10:34:29 AM »
In my more cynical moments, I imagine that if DXO gave more glowing reviews of Canon lenses, the opinions of many people on this forum might change.  ;D
I shoot with a Camera Obscura with an optical device attached that refracts and transmits light

Larry

  • PowerShot G1 X II
  • ***
  • Posts: 71
    • View Profile
Re: Why the DxO bashing?
« Reply #18 on: February 07, 2014, 11:19:45 AM »
Many Canon devotees may also experience sour gripes …

Grapes make wine.

Gripes make whine?  ???

unfocused

  • 1D X
  • *******
  • Posts: 1925
    • View Profile
    • Unfocused: A photo website
Re: Why the DxO bashing?
« Reply #19 on: February 07, 2014, 11:51:41 AM »
In my more cynical moments, I imagine that if DXO gave more glowing reviews of Canon lenses, the opinions of many people on this forum might change.  ;D

Probably true. But I have a slightly different perspective.

Whenever I've looked at DXO results, I've been struck by how small the differences are that they are attempting to measure and quantify.

DXO pretends these differences are significant because if they tested bodies and lenses and consistently wrote that for 99% of circumstances, you can't tell the difference. Well...that wouldn't help them sell their software or drive people to their website or give them free publicity through photography forums.

Similarly, camera brand partisans magnify the importance of these small differences because it confirms their biases, regardless of which side they are on. Canon fans see it as confirmation that DXO is biased, Nikon fans see it as confirmation that their brand is better. The reality is that unless your primary subject is test charts, almost none of this matters.
pictures sharp. life not so much. www.unfocusedmg.com

JumboShrimp

  • Rebel T5i
  • ****
  • Posts: 107
    • View Profile
Re: Why the DxO bashing?
« Reply #20 on: February 07, 2014, 12:31:15 PM »
It appears that DPReview has a similar method (?) to that of DxO whereas DPReview gives a percentage outcome to their testing, such as 82% for a "Gold Award", etc. Looks like both folks give a single numerical rating to cameras and lenses, just using different approaches. Do you think that DPReview has more credibility than DxO? Is DPReview inherently more "accurate"?

Rams_eos

  • Power Shot G16
  • **
  • Posts: 25
    • View Profile
Re: Why the DxO bashing?
« Reply #21 on: February 07, 2014, 12:49:27 PM »
I would say we should not mix DXO software with DXO ratings?

I am extremely happy with DXO software (despite the fact they removed Win XP compatibility without mentioning it clearly in DXO 9). I used it very successfully for Canon and Nikon raw processing.

For DXO rating, I am much more sceptical. With a friend we exchange camera (he has a Nikon D7000 and is a great fan of Nikon) and I totally agree that the end result (the image) is more pleasant with Canon that Nikon despite using a “crap “ Canon (specially for night shoots). But this is my subjective opinion.
We recently saw images of a professional event from this outstanding Nikon D800 (by DXO standards), and we both agreed that overall picture quality was very disappointing :-[.
But we don’t have the control i.e. 5D3 comparison.

So end result is what matters and DXO ratings are reflecting only a few per cent of that.
Deciding which camera to use based on that rating is a bit like choosing a car based the shades of colour you can get. It does not guaranty you will be happy using it.

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #21 on: February 07, 2014, 12:49:27 PM »

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3748
  • POTATO
    • View Profile
    • Nature Photography
Re: Why the DxO bashing?
« Reply #22 on: February 07, 2014, 12:56:39 PM »
I cannot help but question the measurement methodology of a group who (either fraudulently or ignorantly) uses biased and misleading summary statistics to put forth claims about camera/lens performance.  That is to say, if you don't analyze your data properly but staunchly claim to be fair and objective, then it is my obligation to question your data collection methods as well, because your entire process is now suspect.  That is what any good scientist does.

DxO has the phrase "Image Science" as part of their logo, but their practices aren't consistent with that phrase.  I'm most concerned by the 'black box' calculation for their summary Sensor Score and Lens Score (methods should be published), and by the fact that they released data which was incorrect, defended it, then subsequently changed it with no acknowledgement of their error.  Also, I'm noticing that the more I delve into their Measurements, the more I find errors (for example, I just looked at the Canon 28-300L measurements and their actuance data shown visually as field maps are ~10% lower relative to the same data plotted on a graph as a profile).

This right here sums up my problems with DXO in a nutshell. They HAVE made mistakes in the past, ignored them, then quietly tried to fix them without a word. That's irksome. I'm not saying that their DR tests (or rather, Screen DR measurements) are wrong...they are just as valid as any other form of DR test, so long as the same method is applied with consistency.

My problem is that DXO does not clearly explain all of their methods, and sometimes their methods seem sketchy. For example, Print DR is not actually measured, it is derived. Print DR is the "measure" (as they call it) used in scoring their camera sensor tests. But it is not a measure, it is a derivation from the actual Screen DR measure. Problem with Screen DR is, they don't actually publish the actual method by which they measure it, so not only do you have Print DR with is derived from Screen DR...you don't really know how Screen DR is computed. That's quite frustrating. (It gets even worse when you download DXO Optics Pro to see how it works, and find that DXO's algorithms result in FAR more noise with Canon CR2 files than ACR/LR, RawThearapy, or any one of various astrophotography tools...makes me even that much more suspicious that DXO doesn't really know how to properly process Canon CR2 files for optimal performance.)
My Photography
Current Gear: Canon 5D III | Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: SBIG STT-8300M | Canon EF 300mm f/2.8 L II

Radiating

  • 7D
  • *****
  • Posts: 334
    • View Profile
Re: Why the DxO bashing?
« Reply #23 on: February 07, 2014, 01:15:23 PM »
In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?

There is little fault in DxO's methodology, except for a few issues I have with them testing outliers in copy variation.

The problem with DXO is DXO's scores are stupid nonsense.

DXO scores are like rating a car by taking it's price, multiplying by it's horsepower and dividing by it's color in binary. Then situations like this happen:


Newbie: My sensor has the highest DXO mark score of any sensor!
Bystander: What are you using the camera for?
Newbie: Low light reportage!
Bystander: You do realize that you picked an 80 megapixel medium format back to do reportage. It delivers images at 3 seconds per frame, and has an iso range of 100-400 and absorbs only 10% of the light of a Canon rebel, that is the worst possible camera for your needs.
Newbie: No it's the best DXO says so!
Bystander: lol

jrista

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3748
  • POTATO
    • View Profile
    • Nature Photography
Re: Why the DxO bashing?
« Reply #24 on: February 07, 2014, 01:25:51 PM »
In various threads around CR, I have noticed quite a few opinions that are not complimentary to the folks at DxO. The various individuals seem to take issue with DxO's methods and conclusions and generally disagree with pretty much everything they offer. Why? Is there some inherent fault with their methodology that would make their conclusions erroneous? (I am neither pro or con on this issue, but would just like some enlightenment.) Do you have any factual basis for disagreement? Comments?

It should be noted that, in addition to all the valid reasoning offered by everyone else, that DXO's scoring system is not just invalid for Canon. It is invalid, period. Because they utilize only ISO 100 measures in several scores, and because they do not FULLY publish all of their algorithms and methodologies, it invalidates the process as a whole. Using DXO to find the best Nikon camera is no better, and the results are just as misleading for them as for any other brand.

For example, Print DR is an EXTRAPOLATION based on DERIVED data based on Screen DR, which itself is determined via an unknown formula...it is not actually a "measure", despite the fact that they claim as much. Print DR is also based on the assumption that images are downsampled to a standard 3200x2400 pixel size. When it comes time to edit RAW images, ALL RAW images are ALWAYS edited at 100% size. It's the nature of RAW. So when DXO says some Nikon DSLRs are capable of achieving more than 14 stops of DR, they are being extremely misleading to people who don't understand what those numbers really mean. There is NO camera that uses 14-bit ADC that can achieve more than 14 stops of DR. Technically speaking, because of required overhead, even if that overhead is minimized, one could never really achieve 14 stops of DR period, you would always attain something just a little less.

Screen DR is the only valid measure of DR that DXO offers. In the case of the D800 and D600, the actual DR is ~13.2 stops, as much as 1.2 stops LESS than the 14.4 one might believe they have according to the Print DR extrapolation. The fact that Print DR IS an extrapolation means that even if someone downsamples their image by a considerable amount to 3200x2400 pixels, then tries to recover that extra 1.2 stops of information, it may not actually exist...Print DR is not a measure. It is derived, and therefor no one can really know for certain if a downsampled image from a D800 or D600 could actually achieve that much dynamic range. From an editing latitude standpoint...how much real-world freedom you have to push exposure around in post, you can only use DXO's Screen DR measure. The problem is, that measure is not shown by default, you have to find it yourself.

DXO claims a scientific approach to measuring cameras. They really fail to provide that on multiple fronts. For one, they don't measure cameras at all...they measure sensors, and only sensors. That fails to factor in ANY other camera features, such a AF system, metering system, body ergonomics, or other camera capabilities that may be as critical, if not more critical, to final IQ than the sensor. Because they do not 100% fully disclose all details of every methodology and algorithm they use, they can't really claim to be scientific (or at least, not openly scientific.) The fact that they clearly seem to have changed some of their algorithms over the years, and sometimes explicitly to cover up errors they made previously, only debases their claim to using a scientific method even more. At the very least, if an error is found in their methodology, it should be clearly explained and not slyly covered up.

Finally, as others have said...trying to reduce such a complex device as a camera to a single score that could theoretically be used for simple comparison of one camera to another is doomed to failure from the get-go....too many variables to consider, especially considering that DXO ignores the vast majority of them (i.e., all the non-sensor factors.) And don't even get me started on their lens tests...thats a debacle that puts their sensor testing to shame! :P
My Photography
Current Gear: Canon 5D III | Canon 7D | Canon EF 600mm f/4 L IS II | EF 100-400mm f/4.5-5.6 L IS | EF 16-35mm f/2.8 L | EF 100mm f/2.8 Macro | 50mm f/1.4
New Gear List: SBIG STT-8300M | Canon EF 300mm f/2.8 L II

Roo

  • EOS M2
  • ****
  • Posts: 220
    • View Profile
Re: Why the DxO bashing?
« Reply #25 on: February 07, 2014, 01:56:36 PM »
Similarly, camera brand partisans magnify the importance of these small differences because it confirms their biases, regardless of which side they are on. Canon fans see it as confirmation that DXO is biased, Nikon fans see it as confirmation that their brand is better. The reality is that unless your primary subject is test charts, almost none of this matters.

+1
5Diii, 60D, Canon 24-105L, Canon 70-200 2.8L is II, Samyang 14 2.8, Tamron 17-50 2.8 vcd, Tamron 90 macro, Tamron 150-600

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2695
    • View Profile
Re: Why the DxO bashing?
« Reply #26 on: February 07, 2014, 01:59:45 PM »
People bash DxO because during the time when DxO results have been shared around more and more widely talked about on the 'Net, they always favour Nikon/Sony.

As a result, all sorts of reasons have been created by the Canon fanbase as to why this is and how DxO is useless but rest assured, if the shoe were on the other foot, people would be lauding DxO.

JumboShrimp

  • Rebel T5i
  • ****
  • Posts: 107
    • View Profile
Re: Why the DxO bashing?
« Reply #27 on: February 07, 2014, 02:22:14 PM »
If nothing else comes out of this discussion, at least we can hope that the folks at DxO will be reading this, too.

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #27 on: February 07, 2014, 02:22:14 PM »

David Hull

  • EOS M2
  • ****
  • Posts: 187
    • View Profile
Re: Why the DxO bashing?
« Reply #28 on: February 07, 2014, 02:41:45 PM »
Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch. ;)

Many Canon Rumors trolls may experience great satisfaction from demonstrating that ~2 stop deficit in low ISO DR by shooting images with the lens cap on, then pushing those images 4-5 stops in post. 

The majority of Canon devotees, at least here on CR, seem to be fairly objective about the issue.  It's widely acknowledged that Canon sensors deliver less low ISO DR than Nikon/Sony sensors.  The fact remains that people buy cameras, not bare silicon sensors.  If low ISO DR is someone's only criterion for judging a camera's performance (and for a very tiny minority of people, that may be the case), they should choose something other than a Canon camera.  But for most people, what matters is the performance of the system as a whole (camera + lenses + flashes + accessories), and that's where Canon usually wins.   As Don Haines is fond of saying, who cares how many stops of dynamic range a blurry picture has?

Many times in the DR debate, those bashing Canon sensors have been asked to provide examples of shots ruined by Canon's 'poor low ISO DR' that would have been saved by those extra two stops.  Personally, I have almost no examples of that situation - in many scenes, the ~12 stops I get is sufficient, and when the scene DR is greater than 12 stops, it's almost always greater than 14 stops, too.

But for those who would like to persist in this debate, I have found a relevant example showing how an extra two stops of DR can keep the sunlit outdoors from blowing out when shooting an indoor candlelit scene...
One of the better "synopsi" of the issue that I have seen in a while.

Rienzphotoz

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3321
  • Peace unto all ye Canon, Nikon & Sony shooters
    • View Profile
Re: Why the DxO bashing?
« Reply #29 on: February 07, 2014, 02:56:09 PM »
Many Canon devotees may also experience sour gripes that those DxOmark base ISO sensor measurements consistently demonstrate Canon's ~2 stop weakness no matter how expensive a body you buy.
Which leads to much knashing of keyboards as staunch Canonites defend their choice to use such gear and providing many valid reasons and lame excuses why it doesn't matter that a $400 entry-level Nikon DSLR has better low ISO performance than any canon DSLR.
Just watch. ;)
But for those who would like to persist in this debate, I have found a relevant example showing how an extra two stops of DR can keep the sunlit outdoors from blowing out when shooting an indoor candlelit scene...
;D ;D ;D those images show how much DR is available in my Nikon & Sony cameras ;D
Canon 5DMK3 70D | Nikon D610 | Sony a7 a6000 | RX100M3 | 16-35/2.8LII | 70-200/2.8LISII | 100/2.8LIS | 100-400LIS | 40/2.8 | 50/1.4 | 85/1.8 | 600EX-RTx2 | ST-E3-RT | 24/3.5 T-S | 10-18/4 OSS 16-50 | 24-70/4OSS | 55/1.8 | 55-210 OSS | 70-200/4 OSS | 28-300VR | HVL-F43M | GoPro Black 3+ & DJI Phantom

canon rumors FORUM

Re: Why the DxO bashing?
« Reply #29 on: February 07, 2014, 02:56:09 PM »