DxOMark Sensor Performance: Nikon vs. Canon

[jrista]

An Intelligent Nikon Missionary,
May be an oxymoron,
But when missionaries start their trolling,
There's less Oxy and more Moron.

+100000000

HAHA, That's GREAT! ;D

I've concluded that the only way to deal with this whole 5d mk2 vs D800 'debate' is to answer with a little 'nonsense' rhyme ;D

Well, when you have radical, yet diametrically opposed totally blind fools on either side of the argument, like Trumpet and Mikael, it is difficult to make any argument that doesn't get twisted and destroyed by either a brazen lack of understanding or intentional misinterpretation. Poetry seems the only viable alternative.

 

[skitron]

...here is 1dx and d800 lifted the same in the shadows and from a contrast rich motive
This is a strait comparison which shows two different sensors signal/noise in shadows and the cameras are exactly the same exposed...

Not taking sides here, but what software was used to do the lifts? Point being all softs are not all going to do it with the same quality. The Canon shot looks like a total mess compared to what I get from 5D3 shots doing similar lift.

 

[jrista]

...here is 1dx and d800 lifted the same in the shadows and from a contrast rich motive
This is a strait comparison which shows two different sensors signal/noise in shadows and the cameras are exactly the same exposed...

Not taking sides here, but what software was used to do the lifts? Point being all softs are not all going to do it with the same quality. The Canon shot looks like a total mess compared to what I get from 5D3 shots doing similar lift.

There has always been a question about the objectivity of that particular sample shot from Ankorwatt. It has been requested on several occasions that he produce the actual RAW images with metadata, and all we have ever received in return is obfuscation and avoidance.

There is no denying that the D800 has a DR advantage at low ISO, but as you've stated...it is rare to actually encounter that kind of noise in a normal lift. When you are DR limited, it can definitely happen, though. There is a key difference between the 5D III and 1D X that might cause the 1D X to experience worse noise, however: The 1D X uses dual DIGIC 5+ chips. When pixel data from the sensor is split and sent to two separate DSPs for processing, additional banding will usually be introduced. The 7D has the same problem. The 5D III, however, uses a single DIGIG 5 chip to process the whole image, so it is entirely possible it is capable of cleaner shadow lifting than the 1D X.

 

[TrumpetPower!]

proper exposure what is that?

Took you long enough, but at least you've finally acknowledged your ignorance. Congratulations! It's the first step towards knowledge.

In the digital realm, a proper exposure is one in which an image of an 18% gray card results in values in the green channel of the raw file of 18% of the maximum value that can be recorded. With a 16-bit file, values range from 0 through 65,535; 18% of that is 11,796.

The red and blue values will be less than the green values in all but the weirdest light. Typically, red will be a bit more than a stop relatively underexposed and the blue will be about 2/3 stop underexposed. White balancing consists of multiplying the figures from the two underexposed channels so they equal the figures from the properly exposed channel.

Digital exposure compensation is the same basic idea, but applied equally to all three channels. One stop of digital push is accomplished by doubling all the figures in the raw file. One stop of digital pull is done by halving all the figures.

After white balancing and exposure adjustments, a gamma curve is generally applied (with 2.2 being typical of most working spaces). Next, if it's a colorimetric workflow, the ICC profile for that camera is applied. Much more common, however, is that this is where the "special sauce" gets applied. Canon calls them, "picture styles." At the very least, an S-curve is applied (on top of the gamma curve) to boost contrast in the midtones (at the expense of highlight and shadow contrast, of course) and different parts of the spectrum are selectively saturated by different amounts. (It's always added saturation because the camera's native color response is so much larger than any working space). After that, whatever other types of additional modifications (more or less contrast, shadow boost, whatever) are up to you.

In scenes with reflective highlights or the light sources themselves, you generally want to intentionally underexpose the image because otherwise those highlights will oversaturate the sensor and one or more of the channels will record the maximum value for those parts of the image. Underexposing allows those values to fall in a range that can be recorded by the sensor; digital push is then applied to most of the image to restore it, but a compressed push is applied to the highlights -- if you didn't compress the boost in the highlights, they'd wind up just as blown as if you had properly exposed the image in the first place.

Scenes like that are so common in typical shooting that manufacturers intentionally design their meters to underexpose by about a stop. The onboard raw development engine that produces JPEGs applies a corresponding amount of compressed digital exposure compensation, and the various popular raw development engines are smart enough to do the same. There are, however, many tools out there that will let you examine the raw data directly should you choose to play around.

(That scene of the shed in my back yard where I moved the camera's exposure bug six stops to the left, that resulted in an almost entirely black exposure with just hints of detail? The one which came out almost entirely free of noise except for a small amount in the deepest shadows after normalizing the exposure? Considering the meter's built-in underexposure, it was really about seven stops underexposed.)

And now you should understand what proper exposure is.

Cheers,

b&

 

[TrumpetPower!]

there are no proper exposure

Well, at least you've made clear why it is that you can't properly expose an image to save your life.

Sorry, if you're really happy wallowing in your own ignorance and incompetence like this, there's nothing I or anybody else can do to help you with your problems.

And, yes. That's exactly what you're doing. All those shots you keep posting of noisy images taken with a 1DX? That's a display of pure incompetence; no competent photographer ever has anywhere near as much trouble with noise on any modern camera -- let alone a flagship model -- as you keep demonstrating with your 1DX. And it's an incompetence born of ignorance -- ignorance of how to properly expose an image using a DSLR.

Why you're so happy with your ignorance and incompetence is beyond me...but chacun son goût.

Cheers,

b&

 

[Skulker]

proper exposure what is that?

Just one of the basics of photography.

If you are not to sure about this there are many "photography for beginners " type articles on the Web. You could start by trying to understand them. ;D

Your lack of understanding is probably why you can't take a decent photo. I never get the sort of noise in my photos that you get with your canons. Your last comparison showed crazy amount ts of noise and banding etc in the Canon shot and a hopeless soft nikon image. So may be you don't understand focus either and that's why you put up such rubbish shots from both you cameras.

 

[Aglet]

I had a look at the pram photos again and frankly I like the pram on the left. The one on the right is a bit - flat! No character. Just plain dowdy...

Uhmmm.. I think it's an awning, not a pram.

It's also a reasonable choice of scene for such a comparison push-test, with sunlit areas of near-white masonry in the background contrasting with the darks shades of a (nearly) black awning in the shade with its upper side illuminated and the underside in yet deeper shadow. Not a super-high DR scene, but one with enough DR to be able to demonstrate the difference in sensor system noise levels.

I think Ankorwatt is being generous by comparing a 1Dx to a D800. Had he used a camera below Canon's current flagship, the difference would have been even more significant. As it is, the Nikon was half the cost of the 1Dx.
It could have been an even bigger cost/performance difference if he'd have used a Nikon D5100 compared to a Canon 7D.

 

[Aglet]

proper exposure what is that?

Just one of the basics of photography.

If you are not to sure about this there are many "photography for beginners " type articles on the Web. You could start by trying to understand them. ;D

Your lack of understanding is probably why you can't take a decent photo. I never get the sort of noise in my photos that you get with your canons. Your last comparison showed crazy amount ts of noise and banding etc in the Canon shot and a hopeless soft nikon image. So may be you don't understand focus either and that's why you put up such rubbish shots from both you cameras.

Skulker, your attempt at humor is understandable, if only in a locker-room-banter sort of way.
If you have been around long enough to know the poster's true credentials you might realize that sort of comment is better kept to yourself, despite the cute emoticons.
Seriously. There's far too much low-level insult thrown about in the guise of humor instead of intelligent discourse in these contentious topics. Frankly, IMO, I find far too much of a potentially good thread is often taken up by off-topic and unconstructive posts with an inflammatory tone, making it tedious for those who want to follow it without the playground antics and making it more difficult for those such remarks are directed at to remain civil and engaged.
I hope you won't join the ranks of the rancourous on a regular basis.

 

[Sporgon]

proper exposure what is that?

Just one of the basics of photography.

If you are not to sure about this there are many "photography for beginners " type articles on the Web. You could start by trying to understand them. ;D

Your lack of understanding is probably why you can't take a decent photo. I never get the sort of noise in my photos that you get with your canons. Your last comparison showed crazy amount ts of noise and banding etc in the Canon shot and a hopeless soft nikon image. So may be you don't understand focus either and that's why you put up such rubbish shots from both you cameras.

Skulker, your attempt at humor is understandable, if only in a locker-room-banter sort of way.
If you have been around long enough to know the poster's true credentials you might realize that sort of comment is better kept to yourself, despite the cute emoticons.
Seriously. There's far too much low-level insult thrown about in the guise of humor instead of intelligent discourse in these contentious topics. Frankly, IMO, I find far too much of a potentially good thread is often taken up by off-topic and unconstructive posts with an inflammatory tone, making it tedious for those who want to follow it without the playground antics and making it more difficult for those such remarks are directed at to remain civil and engaged.
I hope you won't join the ranks of the rancourous on a regular basis.

I think you'll find 'the posters' true credentials in dispute here on CR.

I see he has acquired himself a 1Dx - lets see some pictures posted in the 'anything shot on a 1Dx' thread.

( And I'm puzzled by someone who repeatedly demonstrates pushing the 8 bit CR.2 file instead of converting it to a 16 bit TIFF first ).

 

[rpt]

As my posts were not constructive, I have removed them.

 

[Loswr]

Frankly, IMO, I find far too much of a potentially good thread is often taken up by off-topic and unconstructive posts with an inflammatory tone, making it tedious for those who want to follow it ...

Many potentially useful threads have been derailed by the very thing the that instigates the 'humor' or outright attacks that lead to thread moderation/locking. I'd say the inflammatory posts are those posts that are made knowing exactly what outcome will result. Before ankorwatt/Mikael had his Mikeal account and all his posts deleted, we'd seen that awning comparison in at least 10 threads, and countless QPcards on barbecues, sheds, and now pianos.

Go to an astronomy forum, and in any thread which mentions the solar system, post a modern diagram of ours and state that Copernicus was right and Ptolemy was wrong. That's just stating a fact, but after a while its going to start pissing people off...and that's not even considering the fact that there aren't many devotees of the Ptolemaic model.

Maybe there are people on Nikon forums who repeatedly post the same example images to show how certain of Canon's lenses are better than the Nikon equivalent. If so, that's just as inflammatory. Should Mikael be insulted for posting the 'truth' that the D800 has a wider DR? No. But repeated posting of the same contrived examples is the epitome of tedious, and downright annoying. Since he's repeatedly posting those examples in response to those he's argued with on many previous occasions, he knows exactly the response it will provoke. That he does it anyway, again and again, and indeed seems to have created new accounts just to do so, says a lot about his character.

 

[TrumpetPower!]

with what should I answer for example trumpet power when he declare that 5dmk3 and 1dx has the same exposure latitude that d800 ? = large DR?

Mikael, I have never, ever claimed any such thing.

My point is, and always has been, that, though the D800 offers marginally more dynamic range, the dynamic range of the 5DIII and 1DX is already most substantial. When you can take a 5DIII image that's been so grossly underexposed that it's mostly black on the preview window and easily rescue it into a clean file suitable for extreme enlargements, it's meaningless to bitch about how shitty its dynamic range is -- which is basically all you do here. It's as meaningless as bitching that minivan x is crippled because it "only" has a top speed of 95 mph, and we know this to be true because minivan y has a top speed of a whopping 105 mph.

So who cares that you can rescue even more grossly underexposed images with a D800?

If you can't get clean images in bright light -- and all you do here is post very noisy images taken in very bright light -- then the problem isn't even close to being with the equipment. The problem is the gross incompetence of the person operating the equipment.

Lots of people here have posted examples of very clean images taken with the 5DIII and 1DX in extreme situations. And those same people have tried to explain to you the basic, introductory-level photographic techniques they used to achieve those results. Yet you continue to insist that those people haven't a clue...while you go on to post yet more really bad snapshots plagued with all sorts of technical problems, all the while insisting that you know more about technique than those who don't have your problems.

Dude, the problem isn't your gear. Your gear is just fine. The problem is with the person who's operating your gear.

Cheers,

b&

 

[Loswr]

Neuro
with what should I answer for example trumpet power when he declare that 5dmk3 and 1dx has the same exposure latitude that d800 ? = large DR? well, I show it with a comparison from 1dx and d800 ,

Do you think your comparison convinced him? Since he's seen it at least a dozen times and still responded the way he did, it's pretty obvious that showing him the same comparison a 13th, 20th, or 200th time won't convince him, either. Did you notice that I've given up responding to your comparison posts? It became pointless.

There's a sign over the bar in The Squealing Pig, a pub near where I work, taken from a Robert Heinlein quote: "Never try to teach a pig to sing; it wastes your time and it annoys the pig."

So, I'd suggest that you simply don't answer at all. I'd suggest the same to TrumpetPower. The issues are important to some people here, but the repeated arguing is really damn pointless. Neither of you will convince the other, just accept it and move on.

</rant>

 

[Aglet]

...
..
So, I'd suggest that you simply don't answer at all. I'd suggest the same to TrumpetPower. The issues are important to some people here, but the repeated arguing is really damn pointless. Neither of you will convince the other, just accept it and move on. </rant>

That is good advice all around.
I think that too often people are compelled to post something in response to a post, even if it's not contributing to the actual technical aspect.

If your momma never told you, "If you don't have something nice to say, don't say anything at all." ..
Well, here it is.

Amending it for an internet based discussion forum would likely go something more like,
"If you can't contribute something useful to the topic, please reconsider posting anything at all."

I have to commend all involved that, at least from what I've seen recently, the level of rancour amongst differing viewpoints has taken a more civil route, even on this topic which has been beaten to death since I joined up a year ago to discuss this very issue.

So thank-you, various posters, for doing your best to remain civil. It improves what is already one of the better places for photographic discourse.

Now for the tech contribution to the topic: I think we'll find that, as Canon moves into DiGiC 5 products, we'll see somewhat less of the general noise and banding issues which plagued a few of the DiGiC 4 equipped models and may have been less of a problem with earlier DiGiC 2 & 3 bodies.

A recent chat with a Canon rep, busy as ever manning a booth, solo in front of a large crowd of curious onlookers, who said that there are many improvements in DiGiC 5 for high AND low ISO noise issues... A point he was eager to make when I told him that I'd dumped all my DiGiC 4 bodies due to poor noise performance for my uses.
He then handed me a 6D and asked me to play with it a while and let him know what i thought of it compared to the 5D2 I had... I know he already knew the answer to that.

Gotta say, other than the reduced overall noise and noticeably reduced FPN on the 6D, the thing I really like about it is the smoother quieter shutter. The 5D2 had such a crude THWACK! along with significant body shake induced by that big mirror being let loose and poorly damped. The 6D feels more like a nicely balanced match-rifle than the shotgun-like 5D2. it's even nicer, i think, than the quick CRUNCH sound the D800's shutter makes.
I haven't had a chance to really shoot with a 6D, likely won't unless I buy one. Likely won't do that either until I find a compelling enough reason to go back to Canon for FF. And about the only reason that will be is if my hankering for a tilt-shift 24mm is not satisfied by the upcoming Samyang for F-mount.
Plunking a 6D on the back of Canon's fabulous 24 and 17mm TS lenses will likely be the most practical solution for some of what I want those lenses for. This might have been speeded up, somewhat, if the darn resale value for D800s was better. All those factory refurbs have dropped my resale value to the point where I now will likely have to keep using both of them for at least a year to get more value from them.

 

[bdunbar79]

See. Nobody cares.

 

[RLPhoto]

See. Nobody cares.

Yup, especially when no photography is in the posts.

 

[sarangiman]

I see the difference in noise and it sure is there! But, this example is flawed in my opinion since the exposure in these photo's was just wrong. I have seen that current Sony sensors offer better shadow recovery, no doubt about that. But in this particular situation it was not necessary to have such an underexposed image and the same photograph could be achieved by both Canon and Nikon camera when properly exposed.

But in another scenario, this exposure could have been needed to avoid clipped highlights. So what you're really complaining about is that I didn't - in my infinite free time during which I happen to have both a D800 & 5DIII on my hands - find a scene with enough DR. That's a fair point, sure, but also - as you can hopefully see - a bit unfair. Because the end result would be the same.

And, frankly, that's what I'd say to everyone here who, in all the comments that followed my original post, indicated that my shot was underexposed. Meanwhile, just a few minutes before this scene, the sun was high enough in the sky that the sky would have blown. So are you complaining that I didn't shoot just a few minutes earlier when there was more scene DR? Would that have changed the end result? Are you arguing that there do not exist scenes that require more DR than what Canon sensors can provide?

Furthermore, according to my histogram when I was shooting, my channels were clipped in the sky! Therefore, the camera led me to believe that I did accurately 'expose to the right and for highlights'. So this is the result an informed photographer would have gotten anyway. Now, if you're going to complain that I didn't have a computer with me to actually check the RAW values as I was shooting, then...

There are many valid suggestions you could make. *Perfectly* nailing exposure within 1/3 EV to get your highlights just short of clipping is not one of them (though it is a noble effort). Not the least because most cameras don't accurately tell you when you're just short of clipping in RAW - in my experience, they tell you you're clipped well before you're actually clipped.

Now, you could have told me to bracket, or bracket for HDR. Valid points - but sometimes that's not an option for certain types of photography. My point is: the conclusion doesn't change. D800/Sony sensor performance gives you much more DR and much more processing latitude. That's useful to some people in certain scenarios.

Also from http://www.luminous-landscape.com/essays/on_safari.shtml#suckout :

The bottom line is that the vast majority of the tonal information that a sensor can capture lies on the far right side. In fact, usually 75% of the total tonal information a sensor can capture lies in the small right hand area of the two top F/stops just below pure white. And yet, almost all cameras leave the factory calibrated to center the histogram instead of moving it as far to the right as possible. To make matters worse, the screens in the backs of cameras are also calibrated to show a good exposure with the histogram centered. This is nonsense!

So what are the consequences of Tonal Suckout? As the name implies, it is as if someone grabbed your image and sucked the tonality out of it, If there are millions of shades of green in a landscape, you may end up with a few dozen. If there is a terrific richness of tonalities in a face, you will end up with a small sub-set that makes the face look bland.

Maybe useful to keep in mind the next time you want to get an underexposed image!

Just FYI, that's actually a fallacy. Read Emil Martinec's extremely informative treatise on noise: http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/noise-p3.html#bitdepth

Essentially, any extra 'tones' will oversample shot (statistical) noise. Consider a signal of 200 photons generated ~100 e- of signal (QE = 0.5). At ISO 100, say we bin 4 electrons 1 DN (digital number, or increment, in the digital file). So that signal is recorded as 25 in the RAW file - pretty darn low (dark). Now, b/c of shot noise, that signal already has a stdev of 10 e- (sqrt of 100), or 2.5 DN. So there you go, you have enough bit depth to still sample the noise in the signal. So it's not limited bit depth/tonality that's the problem - it's the amount of light itself. You expose to the right (ETTR) to minimize statistical/shot noise.

Or to avoid running into the noise floor of Canon sensors... make sense?

 

[sarangiman]

You are still missing the point of argument here, though. Yes, it does work well to underexpose those cameras by four stops, then lift the shadows in post. But doing so is an unrealistic test from a real-world standpoint. All it tells you is that IF someone were to accidentally expose their scene incorrectly by a HUGE amount (some 16x incorrectly), then they would have a greater ability to recover. Purposely underexposing by four stops for the purpose of comparing cameras is also an unrealistic real-world comparison.

Tell that to Sony, who in their A7S is introducing S-Log2 gamma curve application that raises shadows 3 stops and rolls off amplification in highlights. In other words, ISO 800 in shadows, but ISO 100 in highlights. All software-based, since actual ISO amplification off the sensor remains at ISO 100 levels.

If that's a reasonable push for videographers, why not for still photographers? Why is +4EV suddenly entirely outlandish, when a camera manufacturer is now allowing you a +3EV push as an option in-camera?

The levels to which my post was misconstrued was amazing, which is why I chose not to come back to this thread (until now). I mean, I even admitted I still shoot Canon and work around its limitations... but boy have I been so much happier with the Sony A7R sensor (barring shutter shock problems) over the past 6 months. On a boat ride in Cambodia at sunset I set my exposure for the sky and shot away without a care for my exposure - raising shadows 5 EV in post to balance out the shadows of detail & people's faces in the boat (all while holding back the exposure in the sky).

The point stands: +4 EV shifts are not unreasonable. It's the difference btwn ISO 100 and ISO 1600. If such pushes were unreasonable, why would Magic Lantern have implemented a dual ISO feature that people use to shoot ISO 100 and ISO 1600 simultaneously?

Oh, incidentally, Magic Lantern's dual ISO feature wouldn't even be needed on a Sony sensor - because you could perform the ISO 1600 push digitally! Which was the whole point of my original post.

I do still shoot Canon professionally for weddings/engagements, but that doesn't mean I don't wish it had the Sony A7R sensor. It's common knowledge now that some aspects of Canon's sensor tech are severely outdated. ChipWorks stated:

Of the Canon DSLRs analyzed, the imaging chip has remained analog, with Analog Devices’ analog front end (AFE) chips handling A/D conversion en route to the Digic-branded ISPs. Perhaps the column-parallel ADCs favored by others can’t be implemented using 0.5 µm design rules, but more likely Canon is satisfied with its system design and performance.

http://www.chipworks.com/en/technical-competitive-analysis/resources/blog/full-frame-dslr-cameras-canon-stays-the-course/?lang=en&Itemid=815

There are even other, far reaching implications of low base-ISO DR. It limits your ability to retain DR while shooting low light scenes (since you have to raise the ISO amplification in camera, as opposed to selectively raising it in post-processing from an underexposed, lower ISO file).

As for people who don't run into these problems much - YMMV based on the DR of the scenes you shoot. Posting an example of a scene you think has high DR, but where you didn't see offensive noise in shadows, is not a way to prove a camera has just as much DR as another. It's just a way of showing that it had enough DR for that particular scene. Which is why we have controlled tests, like the ones DXO perform.

By the way, thanks to those that stepped in to point out why the misinterpretations of my presentation were flawed. The truth is, low read noise sensor technology will enable new features for photographers only dreamt of in the past. To sit here complaining that some tests push (pun intended) technology too much to show unflattering limitations is, in my opinion, incredibly short-sighted. Which is especially apparent when you begin to see manufacturers and software developers doing +3 and +4 EV pushes of shadows in firmware/software (Sony, Magic Lantern, respectively). That's pretty telling.

 

[jrista]

Hey Sarangiman, welcome back. Way to dig up an OOOLD thread, man! This was years ago...

Before I dive in...which "original post" of yours are you referring to? This is such a long thread, and it went through so many phases...

The point stands: +4 EV shifts are not unreasonable. It's the difference btwn ISO 100 and ISO 1600. If such pushes were unreasonable, why would Magic Lantern have implemented a dual ISO feature that people use to shoot ISO 100 and ISO 1600 simultaneously?

Well, first...I wasn't so much arguing against the use of 4 EV shifts "period". It was more an argument against Mikael's persistent use of particularly extreme "examples" to push is consistently extreme ideas about how uncompetitively bad Canon cameras and sensors are and (at least in his mind) would always be. Now that Mikael is gone, I think we can have a far more reasonable discussion on this subject. I fully agree that more DR is not a bad thing, and in many circumstances (sunsets being an ideal one) potentially extremely useful.

I'd offer that more accurate terminology could be used here, though. The difference between ISO 100 and ISO 1600 is four stops of LOST dynamic range. When you set ISO 1600, you lose four stops of DR, or lose 16x the fineness of levels between your black point and white point, and end up with 16x more noise. I understand what your getting at, though, your thinking about what ML is doing with Dual ISO, however I still wouldn't say that's the difference between ISO 100 and ISO 1600. To be most accurate, it would really be the theoretical difference between ISO 100 and ISO 6. (If you catch my drift).

Sony Exmor sensors still use analog amplification, BTW. When you set a higher ISO, the same fundamental rules apply to Exmor pixels as apply to Canon pixels. You have some charge level in your pixels, you amplify them BEFORE readout, then read (which at that point introduces the read noise). If you don't amplify before reading, even with the D800, noise levels when lifting an ISO 100 shot by four stops are higher than if you just shot at ISO 1600...the difference just isn't as extreme as when you do the same thing with the 5D III.

As for people who don't run into these problems much - YMMV based on the DR of the scenes you shoot. Posting an example of a scene you think has high DR, but where you didn't see offensive noise in shadows, is not a way to prove a camera has just as much DR as another. It's just a way of showing that it had enough DR for that particular scene.

Totally agree. Anecdotal claims are no substitute for proper controlled testing.

Which is why we have controlled tests, like the ones DXO perform.

TL;DR <RANT>

There ARE problems with DXO's results, though. Their direct measures are fine, they are pretty much the only thing of value that DXO produces. The problem with DXO is that not all their "measures" are actually measured, many are derived and extrapolated from actual measures, and they are NOT entirely clear about their exact methodology or formulas. They HAVE been caught on a couple occasions changing their results after the fact when flaws in their methodology were found, without any explanation (at all, not even a blog post or article) of what was changed, how, or why, leaving those who identified or knew about the flaws left to wonder what kind of biased re-weighting or whatever was performed in the black box behind the scenes to "fix" the problems (assuming anything was actually fixed, and that data wasn't simply massaged). DXO has shown considerable bias with their lens reviews, on a very consietent basis, producing very arbitrary results and scores that show certain lenses to be equivalent when they are nothing of the sort, and conversely show other lenses to be inversely related to their actual status (i.e. the EF 50mm f/1.4 lens outscores the EF 600mm f/4 L II lens for the sole reason that it has a wider maximum aperture...results that COMPLETELY undermine the 600mm's vastly superior traits...that clearly indicates poorly balanced lens score weighting that heavily leans towards T-stops as being the most fundamentally important trait as far as DXO is concerned.)

The biggest single issue I think most of the people who complain about them (which does indeed include myself) have with DXO is their often outright and obvious bias, which is exhibited in their scores. If DXO would completely eliminate, drop entirely, the whole notion of linearly scoring the complexity of cameras and camera traits via a single scalar number, I think that would go a very LONG way to improving their reputation among those who hold them in fault. I think that scoring should be eliminated entirely, as I think it is a generally fallacious approach to take when the goal is to produce "scientific" results. When you throw in the fact that many of DXO's scores are weighted, and that some of those same scores are based on "measures" that are not actually measured, they are extrapolated via simple and pure mathematical formulas (and therefor do not actually match reality, when compared with other REAL measures from other testers), DXO is quite simply not as scientific as they claim to be.

That's my core problem with DXO. I like some of their literal measures, ones taken directly from original data. An example would be their Screen DR measure...it's based directly on samples taken from RAW image files. That's a true, honest measure. Print DR, on the other hand, is a purely mathematical EXTRAPOLATION, derived from the Screen DR measure. They call Print DR a "measure", but it is nothing of the sort. Their simplistic and pure mathematic formulas are absolutely no substitute for actually measuring the change in noise levels after actually downsampling a full-sized RAW image to their standardized 8x12" "print" output. The only reason the D800 could possibly score some 14.4 stops of Print DR when downsampled is because the Print DR "measure" is a mathematically computed number, a computation that assumes 100% perfectly ideal circumstances. Reality cannot be ideal, so their formulas are missing aspects of real-world downsampling. If they did actually measure downsampled images, they would very likely produce a less significant difference between the 13.2 stops Screen DR and whatever Print DR ended up being if properly measured, and the differences would shrink as pixel size shrunk (fill factor in FSI sensor designs shrinks with an increase in pixel density, as a greater and greater percentage of sensor die area becomes dedicated to readout wiring and the like...even with reductions in fabrication process, fill factor still shrinks, and when you throw in extra logic onto the sensor die, which is usually the case with smaller processes, fill factor shrinks further).

Sorry to dive into the diatribe on DXO again, but the reasons why there are complaints about them are well-founded, and I think it's important to be clear that they aren't just "because DXO is biased against Canon". I don't really care what the brands are, that honestly doesn't matter to me. There are definable problems with the way DXO does things, because the way they do things allows bias, and bias is quite obvious in many of their scores. The day Canon comes out with a 16-bit ADC and scores 17.5 stops of Print DR on DXO, I guarantee you I'll be making the EXACT same argument AGAINST their Canon scores...because it will be the exact same deal as the D800 and it's 14.4 stops of DR with a 14-bit ADC.

DXO's scores suck. The way they score is very unscientific, regardless of whether it is self-consistent. They should just stick to the raw measurements, they should actually literally measure downsampled versions of each image instead of just running a simple mathematical extrapolation, publish clear and concise facts about ALL of their testing methodologies for EVERYTHING (i.e. be completely open, get rid of the black box), and ditch scoring entirely.

They should rebuild their comparison engine to allow people to pick cameras, and let an openly defined algorithm compute comparative differences and highlight them in a meaningful manner. That would eliminate any option for bias, eliminate this whole inane notion of reducing complex products to single numbers, and potentially educate DXO viewers about the cameras they are comparing, without any DXO opinions sitting in the middle. Just share the raw facts.

 

[Loswr]

Speaking of TL;DR, let me explain...no, there is too much, let me sum up:

DxOMark Scores = Biased Scores = BS.

'Nuff said.