Great review for D800 vs. 5D3 and DxO

[kirispupis]

I like the author's tone here and his opion of DxO. This review pretty much sums up a lot of the current debate - http://www.ronmartblog.com/2012/05/comparision-nikon-d800-vs-canon-5d-mark.html

His style is a bit brash, but I agree with him 95% about DxO (I actually have a more negative opinion of them than he does) and about the entire debate.

 

[sarangiman]

I find it interesting that in that article, the author uses in-camera JPEGs for the ISO comparison (e.g. at ISO 6400) & additionally doesn't downsize the 36MP D800 image to 22MP (which is a fairer comparison). It's pretty obvious that the Canon JPEG has more aggressive NR applied -- the text on the books are not as sharp on the Canon as on the Nikon. I believe a RAW analysis is more telling. Furthermore, DXO scores are normalized, which when combined with the sensorgen.info's extrapolated quantum efficiency for the D800 vs. 5DIII (56% vs. 49%), might explain the higher ISO score DXO gave to the D800.

IMHO, though, I don't see much of a difference in ISO 25,600 performance between the two cameras after the D800 is resized (bicubic) to 22MP (both 5DIII & D800 images were converted using ACR, with identical settings, and please note these images are from Imaging Resource and are not mine):

(Canon 5DIII on top | Nikon D800 on bottom)

View it at 100% here: http://f.cl.ly/items/2k0k1Z402w1s3j3y1W2G/5DIII_vs_D800-ISO25600-2.png.

Maybe a slight advantage to the Canon. So, yes, DXO's 'Sports' score does make me scratch my head a little (the two cameras should at least be equivalent). Perhaps their normalization equation needs to be modified. In video mode, for sure, the 5DIII absolutely trumps the D800, most likely b/c the latter line-skips which, in combination with its smaller pixels = bad high ISO. But that's in video. With images you can effectively bin pixels to get a cleaner image at 22MP. And for those of you who find resizing invalid for comparison -- do you really need to deliver 36MP images to clients?

Furthermore, I also find it interesting that in his examples of shadow lifting, where he concludes that both cameras perform equally, his posted shots are something like 25% & have so much JPEG compression that I can't come to any conclusions at all. Fred Miranda's comparison, on the other hand, showed 100% crops that used high enough quality JPEG compression as to not show compression artifacts.

So while his comparison may be totally valid by some standards, I would caution you to make your own conclusions/comparisons if this 5DIII vs. D800 issue is a big one to you. I personally was also pretty skeptical of DXO... that is until I started performing my own Stouffer Transmission wedge tests & derived similar DR numbers. More importantly, these numbers matched my real world control scenarios between Canon & Nikon sensors. But the whole reason that scientific testing methodologies are developed to begin with is to try & remove variables for objective comparison. This is what DXO does. Again, I do believe the author is correct to raise an eyebrow at the ISO scores given to the 5DIII vs. D800, but I'm willing to bet this is some fault of their normalization equation. Also, it does irk me that DXO doesn't fully publish their full testing protocol, as most scientific journals would require you to, so I'm not saying they're a holy grail or anything. For example their 70-200 f/2.8L II analysis was entirely unreasonable, most likely due to a bad copy (which they themselves admitted). But to conclude that the shadow recovery at ISO 100 between the D800 & 5DIII is pretty much a wash... well... after all the evidence to the contrary...

Also, it should be pointed out that it's a fallacy to think that extra highlight headroom means there's some magical extra DR you can recover from Canon vs. Nikon files. Highlight headroom is a direct result of how the manufacturer's map the data from the sensor to the file and, yes, I have found that a D7000 blows channels 2/3 a stop earlier than a 5DII & 1/3 of a stop earlier than a 5DIII, but the overall extra DR of the D7000 comes from the recoverable shadow detail, which is significantly cleaner than Canon's. Same goes to comparisons of the D800 vs. MF sensors -- the D800 requires a different shooting philosophy whereby you purposefully underexpose the image, and then pull detail out of the shadows. Back to Canon vs. Nikon: if the 5DIII has an extra 1/3 stop of headroom, you'll want to push the actual exposure by 1/3 of a stop (compared to the D7000), but even then your shadows won't be as clean as the D7000 exposed 1/3 stop less. My point is: to do a valid DR comparison, you need to find the overall number of steps any given sensor is able to resolve on the wedge, where SNR>1 (or whatever arbitrary value of your choosing).

Finally: the author complains about the D800 Live View implementation hampering his ability to focus properly & that that may have led to Canon winning the Seattle skyline shot from Kerry Park. Why didn't he use contrast detect focus in Live View? When I do comparisons, I actually specifically use contrast detect focus in LV b/c it's arguably the most accurate way to focus as long as your subject has some contrast (albeit on Nikon you'll have to remember to do that with the aperture wide open for greatest accuracy)...

But even then, I disagree with his statement that Canon won in that Kerry Park shot comparison... I downloaded his full-size JPEGs, and if you look at 100%, it's pretty clear that the Canon shot has had significant sharpening applied to it whereas the Nikon JPEG has not (you can tell b/c the Canon shot has sharpening halos all over it; none on the Nikon shot).

So then I downsized the D800 shot to 22MP, & then applied 176 Unsharp Mask in PS (Radius 1), and 108 Unsharp Mask (Radius 1) to the 5DIII... to get them looking about equivalent sharpness to my eye in the buildings... the Canon needed less sharpening b/c it'd already been sharpened, & even after all this, the Canon file suffers from more sharpening artifacts, yet resolves less than the D800... and that should be no surprise. It, frankly, surprises me that anyone could come to the opposite conclusion. I also tweaked brightness/contrast to get the two images to look similar, for easier comparison.

See for yourself: 100% views, D800 on left | 5DIII on right:

View it at 100% here, paying special attention to the detail in the areas circled in red: http://cl.ly/GVCI/D800_vs_5DIII-KerryPark_RonComparison.png

A bit long-winded, but I find his conclusions questionable, to say the least, in light of all these factors. One must be extremely careful to not walk into a comparison with any inherent bias (hard to do), lest results be skewed in favor of the original hypothesis. It doesn't seem like the author had this intent/bias, so honestly I'm a bit surprised at his entire article.

I shoot with a Canon 5D Mark III, and have been with the 5D line all the way back to the EOS-3 (haha), by the way. But boy have I wondered about switching, lately...

 

[YellowJersey]

I know this has been said before, but I really don't understand why the D800 images have to be downsized to 22mp to match the 5D3 when comparing the two. Surely, people who shoot the D800 are going to be shooting at its max resolution, so downsizing seems to lack real world practicality. I realise that comparing a 36mp image to a 22mp is more difficult, but surely D800 users aren't going to shoot at 22mp often, if ever.

 

[spinworkxroy]

On the recent review of the D800 done by Digital Camera World Magazine, this is a magazine that i read very often because of their unbiased reviews towards any brand and since they're a magazine, they have readers of all brands and they do great tests and even tell you how they do it.
In issue 126 (UK Edition), they did a RAW and JPG Noise as well as Dynamic Range test between the 4 cameras (D4, 5D3, D800 and D700)

And yes, they use DXO software for their tests….
In summary, their charts show the following (as well as their review)
1) D800 is better in DR in raw than the 5D3 up to ISO800(by 1ev), after which the 5D3 takes over all the way
2) The D800 matches the 5D3 noise in Jpg up to ISO3200, which the 5D3 takes over thereafter.

 

[sarangiman]

Downsizing is the only fair comparison b/c when you go to print, there will be interpolation done (either in software or printer driver) to some size/dpi.

For example, at 300dpi, the 5DIII gives you a ~13"x19" print. The D800 gives you a ~16"x24" print.

If you're printing 13x19, the D800 file will be downsized by the software or printer driver. If you're printing 16x24, the 5DIII will have to upscaled (again, if we're maintaining 300dpi, which is the native resolution of most Canon printers, if I remember correctly).

And if you're printing smaller than 13x19, both would be downsized.

So by downsizing, you're sort of simulating what would happen with most reasonable print sizes. In order to make a fair comparison, you want to see the noise characteristics for equal size prints, yes? So you can either downsize the D800 or upsize the 5DIII. For most print sizes, it makes more sense to downsize the D800. Similarly, DXO normalizes to 8MP.

Another more technical reason in my mind is to counter the 'megapixel myth'. It is often falsely assumed that more pixels give you noisier images due to smaller pixels. While it is true that at the pixel level the SNR is lower for a higher megapixel sensor (all other things being equal), one should keep in mind that once you average those pixels to bring the MP of the image down to that of the lower resolution sensor, you also average noise, thereby increasing the per-pixel SNR of the higher resolution image once it's downsized to the lower resolution. Per-pixel shot noise also decreases once the higher-res image is downsized to the lower res (you can do the calculation yourself for a sensor of X resolution & a sensor of 4x resolution... remember that noise adds in quadrature). Granted, you'll likely never get to the SNR of the lower resolution sensor (for the same generation sensors where all else is equal save for the pixel count/size), especially b/c of extra wasted inter-pixel spacing that lowers quantum efficiency. But, you just might approach similar levels of noise, especially if the read noise of the higher resolution sensor is lower than that of the lower resolution sensor (as is the case for the D800 vs. the 5DIII & the D4 supposedly).

The point is: the higher resolution sensor has the advantage of resolution. So you want to give it a fighting chance when comparing it to the lower resolution sensor, which already has the advantage of higher per-pixel SNR, if you're comparing overall SNR. Which is what we're doing, when we're comparing noise levels.

Now, at very high ISOs, I would expect the higher resolution sensor to fall apart, b/c the SNR will drop below some critical value for very low exposure.

And that's exactly what we see in DXO's tests: at ISO 800 & above, even the print (normalized) DR of the D800 drops below that of the D4. I wouldn't mind seeing a SNR 1% or lower curve, which'd highlight such differences even further.

 

[LetTheRightLensIn]

I like the author's tone here and his opion of DxO. This review pretty much sums up a lot of the current debate - http://www.ronmartblog.com/2012/05/comparision-nikon-d800-vs-canon-5d-mark.html

His style is a bit brash, but I agree with him 95% about DxO (I actually have a more negative opinion of them than he does) and about the entire debate.

He doesn't really know what he is doing.

 

[dswatson83]

I'm not a huge fan of the re sizing for comparison either. It makes it easier to compare but it creates a false impression. In daily use, the D800 will be used at 36MP so it needs to be tested at 36MP. If we only care how it performs at 22MP, then Nikon should have thrown in the 24MP sensor which would have made it faster with smaller file sizes (which I wish they did btw). I want to know at what ISO my 36MP image falls apart so I know what to shoot at. Why the hell would I care when a 24MP image falls apart when i'm shooting at 36. Why would I want to know how the dodge viper would run with a v6. I'm sure it would give better fuel economy and a more relaxed ride, but I bought it with a v10, i'm using it with a v10, so I want to know how it handles with a v10.

 

[sarangiman]

As I explained in my previous post, this sort of normalization process is used for (1) a fairer comparison & (2) because ultimately, the print matters. Or, if you're concerned about web-sized output of your images, then downsizing is even more relevant, since both 5DIII & D800 will be substantially downsized for web-based output. The point being, for most types of output, you will be downsizing the 36MP image, save for large prints.

If you would suddenly start making much larger prints just because you bought a D800, then, yes maybe you'd like to do a 100% pixel-level comparison.

If your workflow would include the same size prints regardless of if you get a 5DIII or a D800, then the only fair comparison is the downsizing comparison (which, as you mentioned, has the added benefit of actually making the comparison easier)... or upsizing if you want to print really large.

The point of the downsizing comparison is to compare the noise/detail levels of the cameras if you to print images from them at the same size (e.g. 13x19 @300dpi). With the higher megapixel camera having the added advantage that you could print larger if you wanted to, albeit possibly with more noise at the larger size (vs. less detail if you were to print the 5DIII image at the larger size).

 

[Fishnose]

About the best thing he says in that article is

"Both Canon and Nikon have their strengths and weaknesses so switching platforms will rarely make economic sense. The learning curve from switching Camera platforms can be a big hassle as well for some."

Quite simply, both are bloody good. A little different of course, but bloody good.

 

[psolberg]

Downsizing is the only fair comparison b/c when you go to print, there will be interpolation done (either in software or printer driver) to some size/dpi.

yup. I don't even pay attention to any review showing pointless 100% crops. who da heck sells a 100% crop of a screen shot?. When you print, you normalize and the ISO advantage on the MKIII gets cut significantly when that happens. Ultimately it is such waste to shoot either camera at high ISOs with crippled DR and horrible noise. ISO 100 FTW baby!