Why so much trust in DXO.

[nitsujwalker]

My conspiracy theory:
I wonder if DxO is purposely picking the 8MP downsize option to boost Nikon scores. All nikon cameras that receive great scores have all had multiples of 8MP as the sensor output resolution. The 16MP D7000, 16MP D4, and 36MP D800. Since the DR calculation that Dxo uses is not based solely on the ratio of light to dark but only those values where a signal to noise ratio is below 0dB. I wonder if by perfectly dividing those numbers they are able to achieve a lower noise floor in the shadows enabling larger dynamic range numbers on paper. Notice the random Pentax K5 extremely awesome DR rating also contains a 16MP sensor. I also find it interesting that the $800 16MP Nikon D5100 has an equal DR to the Phase One...hmmmmmm. The Sony NEX-7 with its 24MP sensor (multiple of and the 24MP D3x, 24MP Sony A580,  also have top spots to both $40,000 medium format cameras known their dynamic range and all Canon bodies.

Can it really be a coincidence that EVERY top body in DR happens to have a MP count that is a multiple of the 8MP that DxO mysteriously uses for all of its calculations?
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Ratings/(type)/usecase_landscape

My math may be wrong (never has been my strongest point), but I don't think 36 is a multiple of 8...  Please correct me if I'm wrong.

[AmbientLight]

Quite frankly I don't trust any claim by DXO that certain sensors or lenses are better than others. It looks to me like their results are fabricated more than they are measured. It does not help that their results do not compare well with other published test results or the reality found by many photographers. I also have severe doubts to their claims of being trustworthy. They are a company out to make money. This has nothing to do with providing trustworthy tests, better examples of which can be found at DPReview.

[Canon-F1]

My math may be wrong (never has been my strongest point), but I don't think 36 is a multiple of 8...  Please correct me if I'm wrong.

oh it is... 4.5 x 8 = 36   

[nitsujwalker]

My math may be wrong (never has been my strongest point), but I don't think 36 is a multiple of 8...  Please correct me if I'm wrong.

oh it is... 4.5 x 8 = 36   

Oh silly me... Here I was thinking 'multiple' meant 'whole number multiple' :p 

[skitron]

It looks to me like their results are fabricated more than they are measured. It does not help that their results do not compare well with other published test results or the reality found by many photographers. I also have severe doubts to their claims of being trustworthy.

Actually, I find their charts to be very informative and definitely give me a good sense of what to expect from the copy of lens or body I receive when I buy.

Now the distilled numbers (the so called DxOMarks) are a different story...they really have no practical value beyond generating a bunch of buzz and chatter. Mission accomplished in the case of D800 vs 5D3.

[neuroanatomist]

DR on the D800 is 1/2-1 stop better up to ISO 800, and most have said the Mark III handles colors better. Very radically different from the DxO conclusion ... these tests are flawed in some way.

Not flawed, just frequently misinterpreted.  DxO's conclusions (i.e. overall and use case scores) are based on normalizationto an 8 MP image, and when you do that, you increase DR and decrease noise in proportion to the amount of downsampling.  Thus, a 36 MP image (4.5-fold downsampled) will fare better than a 22 MP image (2.75-fold downsampled).

DxO's measurements, when you look at the non-normalized data, correspond more closely to your statement above, although perhaps the tipping point is closer to ISO 1000 or 1250, but here's what I mean:

[altenae]

It looks to me like their results are fabricated more than they are measured. It does not help that their results do not compare well with other published test results or the reality found by many photographers. I also have severe doubts to their claims of being trustworthy.

Actually, I find their charts to be very informative and definitely give me a good sense of what to expect from the copy of lens or body I receive when I buy.

Now the distilled numbers (the so called DxOMarks) are a different story...they really have no practical value beyond generating a bunch of buzz and chatter. Mission accomplished in the case of D800 vs 5D3.

Well for lenses not.
Glad I did buy the 70-200 version II instead of the better tested 70-200 version I.

[Mt Spokane Photography]

If you have ever dealt with French Companies, you might understand.  They are good people, but they have a different culture and different way of looking at things than we are used to.  We had to require some we worked with to use standard US test methods since theirs gave a answer that made their product look better than the US competition.
 
France, Germany and Europe setup the ISO standards, the whole world belongs, but its over balanced in favor of Europe because there are many small countries each getting a vote.
 
For example, Rumania has 693 Technical committee members while the USA has only 623,(France, Germany, and China have over 700) and its a similar number for the many small countries in Europe where members can often drive from home to attend meetings.  Its due to cost and the USA keeping to SAE and ANSI standards.  Small companies cannot afford to send committee members to Europe for a week or two of meetings when they are outnumbered 100-1 by european countries who want rules that heavily favor European Trade.  Its very Political.  I sent one of my engineers to represent our large aerospace company in a small technical field once, but its a losing effort, and mostly a education into how Europe use the ISO satandards for trade advantages. 

[AmbientLight]

+1 for the comment about French companies. They do have a distinct business culture as does any other country here in Europe.

[Astro]

well if the USA or the UK had to decide over standards we had some crazy S___ standards no engineer with the right mind would want to use.

so i say.... thank good for the ISO committee!!

Mouthful about 1⁄2 fluid ounce (oz.)   

Pony = Mouthful × 2 = 1 oz.

[AmbientLight]

Indeed much of the standardization such as SI metrics do originate in France. Nevertheless this should not be construed to give DXO any credibility beyond that, which they have built themselves.

[neuroanatomist]

well if the USA or the UK had to decide over standards we had some crazy S___ standards no engineer with the right mind would want to use.

What, our systems of weights and measures don't make sense?!?  They're totally logical. Wait...how many drams in one stone?   

[AmbientLight]

Sorry, but British and American weight standards are much like those in antiquity: Anyone can come up with their own weights and anything like a common measurement such as the yard is mere coincidence  . That does not make much sense to me with current societies facing globalization.

I would be in favour of coming up with an international standard for power plugs. Let's use the ones from Jordania, being fully compliant with the simplified European plug design. 

[psolberg]

In the real world, selection bias and brand loyalty tend to be the dominant factors.  The whole point of repeatable tests is to determine, using set criteria, how things actually preform.  When just out and shooting people build a lot of bias into what they think of performance, often with 'this is newer/more expensive, so it must be better otherwise I would be a real fool to have paid so much more for it!' thrown in.

that does not explain how techradar and DXO come to different conclusions... given they both use the DXO testing method.

they don't. they use the same software. there is no way to know if there are any external factors influencing the test. If you take DXO as having test setup free of external interference then it implies that tests that don't match it aren't setup the same or even properly.

You shouldn't trust numbers, trust your eyes.
http://diglloyd.com/blog/2012/20120419_2-Canon5DM3-dxomark.html

Actually.. eyes suck.  The vast majority of what you see is extrapolated, the actual resolution of a human eye is very low, but we have a lot of grey matter behind them that tries to combine the little information it is getting with a bank of world knowledge  and internal models to product its best guess about what it is seeing.   Human vision is very easy to trick and what you expect to see plays a dominant role in what you actually perceive.

are you trying to say something? sorry but I feel asleep half way.....

My conspiracy theory:
I wonder if DxO is purposely picking the 8MP downsize option to boost Nikon scores

If canon had made a 46MP sensor then the conspiracy was that they are helping canon. Canon's score is what canon gets because of the decisions they took long ago, not DXO's.

[AmbientLight]

Reality check, please! I have read here how DXO has set up their tests perfectly and free of external interference and such.
First of all there is no such thing as a perfect test. There is always a possibility for someone doing it better. Secondly I don't believe for one nanosecond that DXO is free of what has been described as Nikon fanboyism. Maybe this affects only one DXO employee determining test results, but that is where it counts, especially if DXO could not get rid of this employee over years. That whole thing smells rather fishy to me, like a fish that has slowly travelled from the Mediterranean sea to the vicinity of Paris, sitting there on a shelf for a while.

If Nikon's market performance would mirror DXO's test results, perhaps there would be something there, but this is obviously not the case. So why are people's buying decisions so different from DXO's test results? Why would photographers owning both Nikon and Canon gear still use Canon gear?

In my humble opinion Nikon's and Canon's products are comparable to some extend and market performance by both companies does support this view, even giving Canon a generous advantage, but according to DXO it appears like no one should buy Canon's products. That doesn't fit reality, so DXO's test results must be way off. It does not help that they try to cover up, by making this appear to be impartial and proper, when it appears to be not.