September 18, 2014, 05:51:07 AM

Author Topic: DxOMark Sensor Performance: Nikon vs. Canon  (Read 80069 times)

Fishnose

  • Rebel T5i
  • ****
  • Posts: 105
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #105 on: September 20, 2012, 05:55:36 PM »

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.   :o

DxO have actually responded to this anomaly on their website when questioned by Canon 70-200mm f2.8 IS II users as to why the apparently superior mark II scores so much lower than its predecessor and their reply was that the older lens has better resolving power (@ 62 line pairs per millimeter vs 52 lp/mm for the new lens). But their website also states that there is enormous variation in resolution throughout the focal range, plus their Resolution Index scores are based on averages of multiple shots taken typically between f5 and f8 throughout the zoom range.

Clearly then, one can challenge the validity & reliability of this 'Resolution' test by conducting a similar test (they  outline in graphic detail their testing procedure) using say multiple samples (half a dozen) of each of the 2 lenses in question (to also test if copy-to-copy variation exists).

Secondly, irrespective of the first answer, what the results suggest given that the Mark II surpasses the Mark I in each of the other sub-categories (especially CA), is that their arbitrary weighting (Black Box is an apt description as they do not divulge their weighting methodology) system is heavily skewed towards 'Resolution' when computing the Overall Score.

The very fact that they place more emphasis on one single category (more so than all other sub-categories combined) is evidence that their overall score is biased & subjective. According to DxO the most impressive Canon lens in the entire EF range is the 85mm f/1.8.

What they should do is post the summary stats of all the resolution test results in a table (freq dist along with standard deviation figures) - that way we can see if it is case of some outlier or skew (to wide or tele) that is causing them to conclude that the mark I is better.

No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

One very good reason that resolution should be weighted quite heavily is that resolution is pretty much impossible to 'fix in the mix'. Okay, a bit of sharpening, but that's not optimal as a solution for lower resolution.

Other issues such as CA, vignetting and distortion, on the other hand, can be fixed very well. And that is precisely the business DxO are in.

canon rumors FORUM

Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #105 on: September 20, 2012, 05:55:36 PM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14390
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #106 on: September 20, 2012, 06:02:35 PM »
Other issues such as CA, vignetting and distortion, on the other hand, can be fixed very well. And that is precisely the business DxO are in.

Sure they can. But not for free...and the price you pay for many of those fixes is decreased resolution.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

sarangiman

  • Canon 70D
  • ****
  • Posts: 291
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #107 on: September 20, 2012, 06:56:43 PM »
Have you tried DPP instead of LR?

Hi neuroanatomist,

Yes, I tried DPP, since threads quickly popped up (I think on dpreview) claiming that DPP fixed the issue. It most certainly did not, IMHO. To me, the default noise reduction settings is what helped cover up the banding. But once you turned all that off, the pattern noise was still there.

Furthermore, DPP allows less ability to lift shadows than Lightroom, making comparisons of more drastic edits impossible.

sarangiman

  • Canon 70D
  • ****
  • Posts: 291
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #108 on: September 20, 2012, 06:59:29 PM »
No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

Yes, it'd be wonderful if they published their full methodology & made their RAW files available. As you mention, that's what'd have to happen in a peer-reviewed journal. I think they might gain more credibility if they did these things.

As for the lens tests -- how many copies of the 70-200 f/2.8L II did they test? The II is most certainly sharper than I wide open on the few copies I've handled. Copy variability can definitely skew results, as has been mentioned before. Not so much the case with sensors, which is why I trust their sensor data.

mystic_theory

  • Guest
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #109 on: September 20, 2012, 07:05:00 PM »

Define capable, and in your definition please address their evaluation of the performance of the 70-200mm f/2.8L IS II, which they score lower than the MkI version of that lens.   :o

No academic journal would publish a DxO report without both Data (in an Appendix) and Methods (with a clearly defined algorithm stating the parameters for weighting each category). In the world of peer review, Black-Box methodology would simply have REJECTED stamped on it and returned.

For a living I don't take pics, but I do publish papers in theoretical physics. Now, compared to the complete lack of rigor in photography testing (at least from my very limited experience of forum, blogs, and other online sources reader), the DxO tests seem outstanding, especially when compared to those of the various Ken Rockwell and company (including DPreview, which now roots for Canon like a cheerleader in the interpretation of their results). Are the DxO tests rigorous enough to be published in a serious scientific journal? Very likely, no. But for what they are meant to do (publicize a software), they are outstanding.

DBCdp

  • PowerShot G1 X II
  • ***
  • Posts: 65
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #110 on: September 20, 2012, 07:45:48 PM »
My 3.2MP Pentax Optio S took wonderful pictures. My Canon 20D was soft over the entire 20K+ range I took with it. The 5D Classic I had was Superb. As is my 5DMkII. But I prefer the images from my old 1DsMkII. What does all the rhetoric actually mean without the burden of proof? Where are the PICTURES?!? 8 pages of hyperexented talk about this or that with zero pictures to back or explain any of the viewpoints!

I like the way a Canon body fits my hand. I like the build of a Pro series 1D. I do not like the ergonomics of Nikon. Either one takes pictures. Neither one composes them. Not a single camera out there sets up the lighting for a perfect shot. Who's perfect shot? Well that would have to be the client. Not the guy who buys the camera, but the multiple clients that pay for it! My client last night was tickled to death with the shots of her business taken with Canon's 5DMkII. That's what counts. Does she know about DxO? Does it matter? Does she or any of my other clients know wether Canon, Nikon, Pentax, Sony or Olympus would give them the look they want? Or do they simply trust me to deliver? So in the end isn't it always a subjective issue based on the need and the delivery and NOT anything at all to do with test results in the lab?

I look at DxO. I look at SLR Gear. I also read DP Review's outlook as well as delve through the pages here. And the only thing I can say with absolute certainty is that I'm not getting my Client's wedding album built while doing any of this.

Show Me The Pictures!  8)

sarangiman

  • Canon 70D
  • ****
  • Posts: 291
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #111 on: September 20, 2012, 08:13:15 PM »
For a living I don't take pics, but I do publish papers in theoretical physics. Now, compared to the complete lack of rigor in photography testing (at least from my very limited experience of forum, blogs, and other online sources reader), the DxO tests seem outstanding, especially when compared to those of the various Ken Rockwell and company (including DPreview, which now roots for Canon like a cheerleader in the interpretation of their results). Are the DxO tests rigorous enough to be published in a serious scientific journal? Very likely, no. But for what they are meant to do (publicize a software), they are outstanding.

I agree.

I'm also in the sciences & therefore appreciate that DXO is significantly closer to rigorous testing (at least for their sensors) than most other tests out there. Roger Cicala is also doing a good job on his blog over at lensrentals. Bill Claff does excellent work as well. Emil Martinec has written outstanding treatises. I like that photography does, in many instances, attract science/engineering-minded folk! Incidentally I've been working on a methodology to rigorously quantify AF accuracy & precision (much like what Cicala has started to do, although he's using a different method) as I find a serious dearth of such information on any site (but lots of qualitative statements everywhere, like, 'I feel like this lens focuses 100% accurately!!'). I think problems like the D800 left AF issue should be easily identifiable by consumers; many other such problems go unnoticed, but actually do affect real-world shooting, surfacing as reduced hit-rates of focus. For example, D800s sent in for the left AF issue, as well as some newer D800 bodies probably going through the same calibration center, now exhibit large front focus of the center AF point in relation to the leftmost & rightmost AF points (which agree with each other). This is not something that microadjustment can fix, and is something that'd go unnoticed by most shooters who microadjust using only the center point. When buying new lenses/bodies, it'd be great to know what sort of AF accuracy & precision we can expect from the combo, and it's not unreasonable to expect wildly varying results given what a complicated process AF is (e.g. correcting for a lens' spherical aberration and how it affects the offset the AF system must apply to the phase data off the AF sensors which only evaluate light from the outer edges of the lens, etc.).

The thing is this stuff is quantifiable. And I'm glad someone (DXO) is doing it properly for sensors. I'd like to see it done for AF, & Cicala has made a great start with his tests.

As for dpreview, actually look at what they had to say about Canon & the 6D:

"Overall, though, it's difficult to shake the feeling that the EOS 6D simply lacks the 'wow' factor of its main rival. Whereas Nikon seems to have taken the approach of taking away as little as possible from D800 when creating the D600, Canon appears almost to have gone the other way, removing as much as it thinks it can get away with at the price. The result is the kind of conservative, slightly unimaginative design that's become the company's hallmark. It's still bound to be a very good camera, of course; just perhaps not quite as good as it could be."

http://www.dpreview.com/previews/canon-eos-6d/6

I think dpreview does a good job of trying to remain unbiased.

canon rumors FORUM

Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #111 on: September 20, 2012, 08:13:15 PM »

itsnotmeyouknow

  • EOS M2
  • ****
  • Posts: 238
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #112 on: September 20, 2012, 08:45:33 PM »
I do weddings and portraits with the 5D3 and have never seen any banding issue.

I see banding in my 5DIII images just from having Lightroom automatically correct the vignetting for my 24/1.4 & 35/1.4 lenses. I just try to ignore it.  :'(

Have you tried DPP instead of LR?

I use LR as that is what I have used for years now and it has a seamless route to CS5.  I really want my 5D3 to work for me but in areas where it's dynamic range is challenged, it fails the test.  There are many things it does very well.  But I can't live with having to smudge all the details with Dfine just because Canon cannot get a grip on chroma noise banding at ISO 100 - 640.  When it gets to 800 or above, the mkIII makes its mark.  The only time as a landscape shooter I will need to use ISO 800 is when I have lost my tripod or am too lazy to get it out of my backpack.  Both have happened before.  I should be able to push a little detail out of the shadows without the whole shot being ruined.  That I can do with the D800.  I cannot with the mk III.

PVS

  • PowerShot G1 X II
  • ***
  • Posts: 72
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #113 on: September 20, 2012, 08:47:25 PM »
If you folks are concerned with DR then just go back to film because film is still the king.
Dedicated 35mm scanners are quite cheap these days, you know?

Anyone who claims his gear limitations is what is holding him back from producing stunning images is a liar.

sarangiman

  • Canon 70D
  • ****
  • Posts: 291
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #114 on: September 20, 2012, 09:26:35 PM »
If you folks are concerned with DR then just go back to film because film is still the king.

Not necessarily. For higher acceptable SNR on the low end, digital sensors trump negative film.
http://www.dxo.com/var/dxo/storage/fckeditor/File/embedded/2012%20Film_vs_Digital_final_copyright.pdf

Of course, this also depends on size of film, since larger film formats will allow for more detail to be pulled out of shadows while maintaining acceptable SNR for reasonable sized prints.

P.S. Just for that paper alone, DXO rocks :)

itsnotmeyouknow

  • EOS M2
  • ****
  • Posts: 238
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #115 on: September 20, 2012, 09:32:05 PM »
If you folks are concerned with DR then just go back to film because film is still the king.
Dedicated 35mm scanners are quite cheap these days, you know?

Anyone who claims his gear limitations is what is holding him back from producing stunning images is a liar.

Depends on  the film.  Velvia 50 for example has very low DR.  I use various film in Medium format bodies.  Dynamic Range can be very important or not important at all.  Images from mkIII often have blocked shadows.  You end up with the choice of blowing highlights or using HDR or pushing detail from the shadows.  I prefer the latter if I must because there is only so much detail that you can get from a blown out highlight and I always shoot RAW.  HDR is hit and miss and you can lose too much contrast. 

banding chroma noise isn't acceptable to me, so I have t take action.  The mkIII is capable of taking great images but if there are shadows you want to lift, forget it.  If I had to pixel peep to see it, it wouldn't be an issue.  But I have had had it visible at web size dimensions.  For £3200 it isn't acceptable.
« Last Edit: September 20, 2012, 09:34:30 PM by itsnotmeyouknow »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14390
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #116 on: September 20, 2012, 09:35:19 PM »
...Are the DxO tests rigorous enough to be published in a serious scientific journal? Very likely, no. But for what they are meant to do (publicize a software), they are outstanding.

As I stated, their testing methodology is rigorous, and the data are of high quality.  It's their data interpretation that's flawed.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

drmikeinpdx

  • Canon 70D
  • ****
  • Posts: 307
  • Celebrating 20 years of naughty photography!
    • View Profile
    • Beyond Boudoir Photo
Nice post sarangiman!
« Reply #117 on: September 20, 2012, 09:48:13 PM »
Nice post sarangiman.  It would be great to see a bit more scientific method in the testing and review of DSLRs.

I had to learn about the poor quality of Canon autofocus on my own, because I could find no really useful information on the net, back in the old days - say about 1-2 years ago.

I am currently awaiting the arrival of a 5D Mark 3 with great hopes for my future autofocus happiness.  I've studied Roger Cicala's articles in great detail and I am prepared to buy Canon lenses that use the new autofocus method if I feel the need.  I'm going to try the Mark 3 with my current lens collection first to see how it works after careful micro-adjust.  I'll let folks on this forum know how it works out.

Mike
Current bodies:  5D3, 7D, 550D, S100
Favorite lenses: 135 f/2.0 L, 85 f/1.8 200 f/2.8 L, 50 f1.4 Sigma, 40mm pancake, 24-105 L.
blog:   http://www.BeyondBoudoirPhoto.com

canon rumors FORUM

Nice post sarangiman!
« Reply #117 on: September 20, 2012, 09:48:13 PM »

sarangiman

  • Canon 70D
  • ****
  • Posts: 291
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #118 on: September 20, 2012, 10:31:29 PM »
I think you'll be happy with the Mark III, Mike. Actually Canon's AF as implemented in the Mark III in combo with its 24L, 35L, & 85L primes is the only reason I'm sticking with Canon right now. The DR & Auto ISO implementation on the D800 would otherwise be enough to make me want to switch. Well, and native use of the 14-24 :)

Note that for primes you'll want to microadjust based on your shooting habits. Since I tend to use the primes to shoot rather close subjects (that's more my style), I microadjust using a LensAlign at a distance of 25x focal length. I think manufacturers tend to suggest 50x focal length, as the best compromise. Certainly, using 25x focal length, infinity no longer focuses properly at wide apertures... it's a shame camera companies haven't implemented some sort of interpolation for microadjustment values based on subject distance (like they've done for focal length of zoom lenses). Maybe it's coming?

This does affect me in real-world shooting. For example, the other day I happened to be using my 85L in a non-standard way, shooting subjects more distance simply b/c of the way the event was set up. I noticed I got better results by resetting my microadjustment to 0 (which works best for distant subjects).

And then there are days where the microadjustment just seems to be off... haven't quite figured that one out yet... but luckily, it's not too often.

The good thing about the Mark III is that its precision is good enough that usually you can tell if the lens if front focusing or back focusing. Not so much with my Mark II, which had so low precision that I just couldn't tell b/c focus was all over the place.

Now, I realize I'm placing high demands on the system b/c I'm shooting below f/2.0... but why else buy a prime if not for that almost 3D look of having a subject pop out from a blurred background? That's generally what I'm interested in when I'm using primes... not always, but most of the time.

The Mark III brings me closer to achieving that, without having to take 100 shots just to get 15 or 20 in focus at f/1.4.

The Nikon D800 also has very good (similar to Mark III) precision. But there are other issues with lenses that I won't get into here, but will hopefully write an article about soon. Furthermore, their focus points being totally miscalibrated with respect to each other is just something I don't wish to deal with. Luckily, my 5DIII focus points are pretty consistent (the leftmost one slightly backfocuses compared to the others, but its acceptable and nowhere near as drastic as what I've seen testing 4 different D800 bodies).

bdunbar79

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2571
    • View Profile
Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #119 on: September 20, 2012, 11:17:56 PM »
...Are the DxO tests rigorous enough to be published in a serious scientific journal? Very likely, no. But for what they are meant to do (publicize a software), they are outstanding.

As I stated, their testing methodology is rigorous, and the data are of high quality.  It's their data interpretation that's flawed.

Yes, this.
2 x 1DX
Big Ten, GLIAC, NCAC

canon rumors FORUM

Re: DxOMark Sensor Performance: Nikon vs. Canon
« Reply #119 on: September 20, 2012, 11:17:56 PM »