July 24, 2014, 07:17:48 PM

Author Topic: 24 cameras now pushed and compared for noise on the dark end at all ISOs  (Read 5889 times)

Aglet

  • 1D Mark IV
  • ******
  • Posts: 904
    • View Profile
Did you account for ISO differences between cameras?

For example, ISO 200 on the 5D2 is about ISO 143. The 50D's ISO 100 is about ISO 157. DxoMark has all the numbers.

http://www.adorama.com/alc/article/11227


HI, Thanks for asking.
I gave it only slight consideration as it adds a lot of work to normalize all those test shots using their tested actual ISO. And, since this isn't my day-job  ;)  My reasoning is that pretty much all the decent cameras out there have an actual ISO that should be within 1/2 stop of rated ISO you set them to so the worst case would be comparing one that's a half stop low to another that's a half stop high; and the latter rarely ever happens from what I've seen on DxOmark. My intent here is to show the noise structures of various cameras across their ISO range in a way they can be loosely compared.

More importantly, it is the structure of the noise distribution on the image, not just the total amount of it, that is key to how well your images may hold up in post before the dreaded banding type noise structures put a halt to your intentions.
The comparisons I've put up so far will at least allow people to see for themselves the type of noise these camera sensors are generating on a small scale relative to the whole image.  When I get the whole-sensor noise map up it's another story.  Larger noise structures may then show up, as they do on the 40D sample near the beginning of this thread where you can see lots of strong horizontal lines.


====

Aglet -thanks for this wonderful job.
I'm a bit surprised somewhere there:
1. 7d looks better than 5d2, what is difficult to be confirmed by me as I use both.
2. I'm surprised with g1x which looks awfull in comparison to g12.

Because of question No 1: have you assured the same conditions during tests? As conditions for Darks I can't imagine anything else as same temperature (also not so, that camera has shot before 100 photos and after that you took a sample dark frame). Were the baterries fully charged? Noise reduction of course Off?
What was the exposure time? Have you tried different exposure times and compare it also in this dimension?

I'm asking just because of observation No 1 - If my experiences from using both show something different,  then how can I trust and interprete the rest?


Glad you find this info helpful or at least interesting. :)

Re: 1. IMO, I don't think the 7D looks better than the 5d2 in this comparison. They have similar levels of average noise at similar ISO settings but the 7D's noise is more coarse and has more obvious vertical banding at low ISO settings.  Vertical banding on the 5d2 is more subtle and cleans up a little better in post with some software.

Re: 2. Yes, G1X is a very different beast with a Canon CMOS sensor in it vs the previous G12 which, I think, uses a Sony-made sensor.  I was actually surprised the G12 was noticeably less noisy than the G11 despite the G12 having more advanced video features on it.

As for testing conditions, all the above cameras were tested at ~21C ambient temperature and were only operating for a minute or so before starting the test shots to turn off any in-camera NR or other toning.  Batteries actually were all right near fully charged for the above cameras.  The set of cameras I added this weekend had battery charge at various levels (beyond my control) and temperatures may have been a little higher as some of them had been operating occasionally prior to my testing.

Exposure times have all been done at 1/200s. An arbitrary choice on my part but picked to be representative of a typical daylight landscape exposure speed; fast enough for the sensor not to exhibit any hot pixels that would add another dimension to the testing.
i have also done some long exposure tests and long exposure with in-camera dark-frame subtraction NR applied.  The results are surprising! :o  i will get around to posting them sometime too.

===


First, pushing shadows by four stops is about as meaningless as it gets. If you were to pull the highlights by an equivalent amount, you'd be attempting to add eight stops of dynamic range to the camera, basically doubling your DR. If you're trying to do that in the real world, either you know full well that the results are going to be crap or you're merging multiple exposures with HDR.

But even if you're only doing this to make the quality of the noise more visible, it's still a meaningless test because you're comparing 100% crops. A 100% crop of a 50 megapickle image can look much, much worse than a 100% crop of a 10 megapickle image and yet the 50 megapickle image might make a print that blows away the 10 megapickle one.

If you're serious about this, what you want is to pick a representative final print size -- 13" x 19" would be typical of the largest likely print size owners of those cameras might have in common -- and then scale, up or down as needs be, all the images to 100 ppi at that size. For this example, that would be 1300 x 1950 (to preserve the aspect ratio). You can then take a 100% crop of the 1300 x 1950 resized image for your comparison.

Why only 100 ppi instead of the 300 ppi you'd be printing at? Well, 300 ppi would make sense if you were going to print these, but you and everybody else will be making the comparisons on screen and not on paper.

Of course, the real way to do this kind of comparison would be with real-world images that get put through the full post-production workflow all the way to final prints, and then compare those prints...but nobody seems to do that sort of thing. I'm guessing it's because those who make prints don't waste their time with these sorts of tests, and those who waste their time with these sorts of tests don't make prints.

Cheers,

b&


Thanks for your feedback but I have to disagree with you on some of your points.  Of course, if you don't know the background to this whole saga I can understand your point of view. :)

First, it's not meaningless at all, unless you don't do any serious adjustments in post, as some of us do. .. And not necessarily as a "correction" for something missed in taking the shot but as a way to help map the maximum dynamic range of our cameras to the very small dynamic range of a print or even a digital display.  You can't multi-shot HDR moving subjects worth a fly-spec, neither should you have to with 14bits worth of camera raw data.

100% crops are useful just to see the small scale noise structure, like the fine banding of the 7D and 5D2 or some of the G-series cameras.

If I wasn't serious about this I wouldn't waste more of my time on it. It's my gift to you, free information on a topic that's so far been poorly addressed and inadequately covered, leaving many confused or uninformed.  I'm presenting it in a way that will hopefully help more people understand the limitations of their gear so they can work around it and be more aware when they can encounter problems between shooting and processing.

I was ignorant of my camera's shortcomings and overly relied on it, being a bit of a fanboy at the time. Then, being unable to process some of my real-world shots for large scale printing, because the banding noise was an issue, is what started this whole project.  However, even little 18x12 inch "snapshots" can show some banding noise structures under some conditions.

I'm tempted to say that anyone who hasn't run into this sort of problem is either shooting with some better cameras or hasn't yet had to learn how to wring the utmost from the equipment they have.
... But i won't. ;)

=====

* Comparing Canon to Sony's sensors (Nikon, Pentax): It is indeed the question how valid such an extreme underexposure and successive +ev correction is, or Canon would be broke by tomorrow. My idea: maybe another test with a dark grey background and moderate underexposure and post-processing +ev correction would put this into perspective? This is because that somewhat clever post-processing (software) will never try to raise black parts as far as you did.


Perfectly valid and that is one of the things I'm trying to devise a good in-studio test for because the black levels and the shadow levels can behave differently.  The tests I've done and displayed certainly do not tell the whole story. But they are a good prolog for generating more awareness that noise structure issues are of some concern under certain conditions.

* To me, the thing that kills my shots is banding. I guess there is no scientific number to measure this? If not, downsized shots would indeed offer a better basis for comparison since they'd show how much noise *pattern* is left.


YUP. That's it and I wish there was some measurement of this.  I haven't found any yet so decided to try cover this topic.
I don't often downsize my shots so for me, pixel-level evaluation is where it's at.
If you're printing smaller then some of these things do get averaged out a bit.

Some cameras, with strong but fine-grained noise and no obvious banding, will not have major issues in post as NR can often mitigate the noise without hurting the actual image too much.  Another camera can have weak levels of noise but a regular banding pattern to it may make it far less amenable to being pushed in post. The comparisons I've put up so far will at least allow people to see for themselves the type of noise their camera sensors are generating on a small scale.  When I get the whole-sensor noise map up it's another story.  Larger noise structures may then show up, as they do on the 40D sample near the beginning of this thread.

* But most of all, I am stunned how much worse the 7d looks in this artificial test in comparison to the 60d. I have heard that the 7d is more prone to banding, but I'd never dared to mention it here because I'd get stomped. But looking at your 7d/60d pictures... am I missing something? They do use the same sensor, don't they? Is this banding due to some other processing in the 7d's dual digic cpus?


7D uses a "dual readout" circuit for faster shooting speeds.  This may be the problem, with each signal path taking alternate 8-pixel wide swipes of data?..  If they're not perfectly matched, then banding errors show up in the final file dark areas.  Fine vertical banding still exists (iso200, 400) on the 60D but it's MUCH more subtle, barely perceptable for the most part. The 5D2 is a bit worse than the 60D.

I've tested a second 7D and 5d2, they were nearly identical with the first samples.  I suspect there may be some 7Ds that are cleaner, some that may be worse. if it weren't for the 8-pixel banding, I'd be pretty happy with mine!

=====
all as expected (well D4 looks better than expected) but it's interesting to see it visually


it is impressive - I hope there wasn't some hidden NR feature that I didn't find and turn off! The thing is almost TOO good to believe.

you can see the nasty 7D vertical banding, it had a special form of it that none of the other canon cameras had, it also varies quite a bit from copy to copy of the 7D and across the frame, with some copies have noticeably more than others overall and most having it vary in degree across the frame


I've only looked at mine and one other and they both exhibit virtually identical noise behavior across the frame but I'd not be surprised there's variations, given the design.

.. i have a weird feeling the 1DX will test closer to the D4..



I certainly hope so.  I was able to play with a pre-production one a little. menu is very similar to the 5d3, overall handling and feel was pretty sweet with decent ergonomics for the 10 minutes I played with it.  If it wasn't for that sealed memory card door!...


One strange thing though is I havent seen numerical data hinting at the D4 being so many stops better DR at high ISO or the 5D3 being that many stops DR better there either. Something seems odd even considering ISO gains are not normalized and MP counts are not normalized.

What software was used to make these? Pre-raw processing software or just ACR which might do different things under the scenes for each?

Anyway interesting, never seen such an extensive comparison of this type before.



All processing done in ACR, so yes, could be some differences in the way it treats various camera files that user controls can't access.  Best tool i've got for the job at the moment tho.

OTOH, the K5, D4, D800 and even the sub $700 D5100 all start off about the same in this race and the $ value starts to sort itself out as you move up the ISO scale.

This is still more of a noise measurement than a signal to noise measurement at this time.
Thus the various suggestions of doing this again at a low exposure value (not = to black) will bring a different context to put this in which will be more representative of real-world images and also may show some different behaviors from the cameras.

It's on the list of things to do. :)


canon rumors FORUM


dr croubie

  • 1D X
  • *******
  • Posts: 1397
  • Too many photos, too little time.
    • View Profile
For instance, have you seriously looked at the Pentax K5?...

Yeah, my mum's got one, and honestly, if I didn't already have a 7D, i'd be seriously considering the K-5 instead.
Weather-sealed kit-lens, in-built timers (because opening the flap for my TC-80N3 doesn't make it waterproof anymore, does it...), auto-iso that actually freaking works with +/- EV Comp, almost all other 7D features at a 60D price.
Frankly, being able to use the magnificent EF-lens-lineup is probably the only plus to buying a 7D over a K-5 imho...
Too much gear, too little space.
Gear Photos

Marsu42

  • Canon EF 400mm f/2.8L IS II
  • *******
  • Posts: 4356
  • ML-66d / 100L / 70-300L / 17-40L / 600rts
    • View Profile
    • 6D positive spec list
For instance, have you seriously looked at the Pentax K5?...
Yeah, my mum's got one, and honestly, if I didn't already have a 7D, i'd be seriously considering the K-5 instead.

I nearly got a K5 (same sensor as the Nikon d7000) instead of the 60d - but the catch is that lenses are much more expensive because of the limited selection and there's a much smaller market for used ones, not to mention the equipment like flashes etc. That's a good reason to stick to Canon or Nikon, although it's not fair towards Pentax because this results in a Catch 22 and them sticking in the same market niche forever.

all as expected (well D4 looks better than expected) but it's interesting to see it visually
it is impressive - I hope there wasn't some hidden NR feature that I didn't find and turn off! The thing is almost TOO good to believe.

You might not be far from the truth there! I remember back when deciding between the d7000 and the 60d I read a review about the raw files of the d7000 - I cannot find the link because I concentrated on Canon since then. But it said that the d7000 applies more *forced* nr to raw files than the d90 to cover up for the increased noise that goes along with the 12mp->16mp transition. The guy who wrote this was a long-time Nikon insider and obviously really knew what he was writing about since he was able to figure out this at all comparing the raw files of the different models at different iso settings.

But this might be an indication that the Nikons have a built-in nr gimmick that is designed to make them shine in artificial tests just like yours. Maybe Nikon sites have more information on this. Graphics chip manufacturers like ATI or NVidia do this all the time when "optimizing" drivers for specific benchmark suites.

If you suspect something like this, too (and the differences between Nikon and Canon are indeed too good/bad to be true) it's really time for a more real-life oriented test.

Aglet

  • 1D Mark IV
  • ******
  • Posts: 904
    • View Profile
I nearly got a K5 (same sensor as the Nikon d7000) instead of the 60d - but the catch is that lenses are much more expensive because of the limited selection and there's a much smaller market for used ones, not to mention the equipment like flashes etc. That's a good reason to stick to Canon or Nikon, although it's not fair towards Pentax because this results in a Catch 22 and them sticking in the same market niche forever.

I had a good chat with my local Ricoh/Pentax Rep, Bill Gouge, a fantastic fellow who works with tireless enthusiasm to promote their product.  He showed me features on the K5 I never even imagined a camera could do; like the GPS-based ability to do astrophotography shots without making star-trails, without needing an equatorial mount. YUP.  The camera's sensor stabilizer mechanism is used to maintain relative image stillness as the world turns. Just amazing! And that's just one of many unique abilities on that body.
I was also told that Ricoh is in the process of working up some ads to promote their lines and they do need to increase mindshare.

I've always been rather ignorant of the Pentax group of products, never knew anyone that had one until more recently in my life.  The Pentax followers I do know now are quick to sing the praises of the platform, like the ability to use many old and easily obtained Pentax lenses from WAY back, sort of like Nikon who'd maintained a great deal of backward compatibility.  Their new lenses aren't necessarily expensive either, actually, I was quite surprised at their pricing.  They've really put a lot of photographer-engineering details into their stuff.  I do wish they had some better zooms but they have some great primes and 3rd party support is there too.

.. the d7000 applies more *forced* nr to raw files than the d90..
- clip -
If you suspect something like this, too (and the differences between Nikon and Canon are indeed too good/bad to be true) it's really time for a more real-life oriented test.

There are multiple kinds of NR processes. Some are desirable hardware methods that reduce noise signal by clever use of electronics. This improves rather than detracts from the end result.
NR done after data-collection is more of a software based approach and can have a negative impact on the signal or image quality.
Without knowing how much of each is applied it's hard to say without more rigorous testing but when it comes to properly designed electronic hardware NR methods, bring it on!  That's the way we get quiet sensors. :)
When the D3s first came out it looked very impressive for a moment but then I noticed the dark levels were artificially crushed and that helped kill noise.  i don't know how it fares now but there IS noise on the darkest of Nikon frames, it's just super low level.  Things are a little different and, ultimately, it's the final IQ that matters.  So I sit at my desk over lunch and daydream of a good test regime to use to compare these things.  :)


ers811

  • Power Shot G16
  • **
  • Posts: 24
    • View Profile
First of all.... THANK YOU!  This is really great free info.

A 100% crop of a 50 megapickle image can look much, much worse than a 100% crop of a 10 megapickle image and yet the 50 megapickle image might make a print that blows away the 10 megapickle one.

If you're serious about this, what you want is to pick a representative final print size -- 13" x 19" would be typical of the largest likely print size owners of those cameras might have in common -- and then scale, up or down as needs be, all the images to 100 ppi at that size. For this example, that would be 1300 x 1950 (to preserve the aspect ratio). You can then take a 100% crop of the 1300 x 1950 resized image for your comparison.

Yes, yes, and YES!

This is what I'd really love to see.  For any shot that's framed more-or-less correctly to start with (I'd say most are), the larger resolution noise is going to be compressed and never look like the 100% crop.

My inner nerd really loves seeing all these comparisons, but the rational part of me that PPs and prints photos would love to see how 22 MP and 36 MP noise compress into the same final print size....

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3284
    • View Profile
For instance, have you seriously looked at the Pentax K5?...
Yeah, my mum's got one, and honestly, if I didn't already have a 7D, i'd be seriously considering the K-5 instead.

I nearly got a K5 (same sensor as the Nikon d7000) instead of the 60d - but the catch is that lenses are much more expensive because of the limited selection and there's a much smaller market for used ones, not to mention the equipment like flashes etc. That's a good reason to stick to Canon or Nikon, although it's not fair towards Pentax because this results in a Catch 22 and them sticking in the same market niche forever.

all as expected (well D4 looks better than expected) but it's interesting to see it visually
it is impressive - I hope there wasn't some hidden NR feature that I didn't find and turn off! The thing is almost TOO good to believe.

You might not be far from the truth there! I remember back when deciding between the d7000 and the 60d I read a review about the raw files of the d7000 - I cannot find the link because I concentrated on Canon since then. But it said that the d7000 applies more *forced* nr to raw files than the d90 to cover up for the increased noise that goes along with the 12mp->16mp transition. The guy who wrote this was a long-time Nikon insider and obviously really knew what he was writing about since he was able to figure out this at all comparing the raw files of the different models at different iso settings.

But this might be an indication that the Nikons have a built-in nr gimmick that is designed to make them shine in artificial tests just like yours. Maybe Nikon sites have more information on this. Graphics chip manufacturers like ATI or NVidia do this all the time when "optimizing" drivers for specific benchmark suites.

If you suspect something like this, too (and the differences between Nikon and Canon are indeed too good/bad to be true) it's really time for a more real-life oriented test.

The thing is that people have shot the same real-life scene side by side with D800/D7000 and Canon cameras and you could pull so much more nice looking detail out of the shadows, the real world difference WAS there.

LetTheRightLensIn

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3284
    • View Profile
I had a good chat with my local Ricoh/Pentax Rep, Bill Gouge, a fantastic fellow who works with tireless enthusiasm to promote their product.  He showed me features on the K5 I never even imagined a camera could do; like the GPS-based ability to do astrophotography shots without making star-trails, without needing an equatorial mount. YUP.  The camera's sensor stabilizer mechanism is used to maintain relative image stillness as the world turns. Just amazing! And that's just one of many unique abilities on that body.
I was also told that Ricoh is in the process of working up some ads to promote their lines and they do need to increase mindshare.

wow that's awesome

at least some companies are being less than boring


I

canon rumors FORUM


dr croubie

  • 1D X
  • *******
  • Posts: 1397
  • Too many photos, too little time.
    • View Profile
The Pentax followers I do know now are quick to sing the praises of the platform, like the ability to use many old and easily obtained Pentax lenses from WAY back, sort of like Nikon who'd maintained a great deal of backward compatibility.


Well yeah, but canon gets the best of the adaptable-bunch, we can use old Pentax M42 (I do), Olympus OM (I do), more Nikkor lenses than Digital Nikon bodies can, C/Y, and Leica R, plus all Medium Format (I do).
Pentax K bodies can only use M42 through good adapters (i've heard chinese ebay adapters can break lens and/or camera mounts), they should be able to use Nikon F but I don't know.
Nikon bodies get boned by only being able to use some old nikkors (that you need a big table to navigate) and only Medium Format above that.

I've been meaning to buy a PK-EF adapter to try out the weather-sealed 18-55 kit lens from the K-5 on my 7D, see how it stacks up against my 15-85 for sharpness etc (maybe i'll borrow my mate's EFs18-55IS to compare too)
Too much gear, too little space.
Gear Photos

Marsu42

  • Canon EF 400mm f/2.8L IS II
  • *******
  • Posts: 4356
  • ML-66d / 100L / 70-300L / 17-40L / 600rts
    • View Profile
    • 6D positive spec list
at least some companies are being less than boring

Indeed, that's just what I was thinking, too when I read this. And it is what I am missing from Canon, e.g. when reading the 5d3 specs. Nikon D800 has face detection and the like, but Canon seems to be quite conservative here except putting in newbie features like the "creative zone" on the 60d. I'm not a pro (yet), and thus I'd like to see more innovation to play with and I like magic lantern - the camera body has a built-in computer, why not use it?

canon rumors FORUM