24 cameras now pushed and compared for noise on the dark end at all ISOs

[Aglet]

I managed to get my mitts on another NINE cameras this weekend and have posted the results along with the previous 15 on one page. A big wide monitor is ideal to look at the whole picture.

Now, not only are there the usual suspects from Canon; 5D2, 5D3, 7D, 60D, 50D, 40D, etc. The G1 X has been added to the PowerShot group.

There are now the new Nikon D4 (seriously dark), and D800 along with previous D5100 and D90.

Also new to the comparison are Pentax medium format 645D, APS-C sensored K5, K01 mirrorless, and even the tiny Pentax Q.
Also Sony A900 full frame SLR and Panasonic G1X four-thirds class camera.

All are pushed +4EV using my usual procedure.

http://a2bart.com/tech/tech.htm


I handled a really neat little Olympus OM D but only a pre-production model so I couldn't get files from it.  It did fit the hand nicely and featured plenty of dedicated controls on a squarish body with a bit of a retro look.  The grip and extra battery (separate units) made nice additions to the basic body.

So, I've had the chance to play with a LOT of cameras this weekend, learned a few things and shook up my imaging world a little more.  For instance, have you seriously looked at the Pentax K5?... What a control-freak's ultimate camera.  It may not get used like the workhorses like the 5D3 are but it has a lot of things that would appeal to the creative shooter.
Pocketable Pentax Q puts out impressive images for such a tiny little unit too and it sure is "cute."  A couple more lenses for it would be nice.

Unfortunately I forgot to get files from the new Fuji X-Pro 1. DANG.

I DID remember to order my next new imaging tool though. (for delivery sometimes later this year)

Well, I don't mean to do a whole whack of reviews here, just notify those who are interested in dark frame noise tests that there's a bunch more to look at now.
It'll still take me a while to post the whole-sensor reduced images to show larger scale noise patterns but some of them are pretty interesting and may reveal a few things going on inside some cameras for those with a serious tech interest.

[nightbreath]

The same issue as it was before, 40D noise looks better to me than 5D Mark III noise at ISO 100 and 200. Is it possible to show 10% of image area rather than 100% crop?

It would give much better understanding on overall camera performance.

[bycostello]

fascinating comparison...

[Aglet]

The same issue as it was before, 40D noise looks better to me than 5D Mark III noise at ISO 100 and 200. Is it possible to show 10% of image area rather than 100% crop?

It would give much better understanding on overall camera performance.

Agreed.
What I'm working on next is showing the whole image, reduced to 1/8 size in both dimensions, which gives a good indication of overall noise patterns and other interesting patterns that can provide clues to what's going on in the camera system.

I'm looking for a different scaling algorithm tho (I think I have a B-spline somewhere) as the usual ones for photographic work de-emphasize the overall luminance values too much so the scaled whole-image noise map is much darker than the 100% crop at a given ISO.  That isn't a big issue either tho, since one scaled whole image noise map at one ISO will look pretty much like any other ISO level for a given camera.  There's generally either repetitive fixed pattern noise (sensor or readout system irregularities) or random banding noise (internal power supply noise) which is never quite the same from shot to shot. 

I've attached some 1/8-samples here, based at 3200 ISO, until I can devise a better procedure for processing and presenting these images from all the cameras. Altho you can still make out the patterns, the actual range of luminance variation within each image is greatly decreased by the scaling and jpg compression.

Also just noticed, the CR posting system here seems to rescale attached images to a max width of 702 pixels so the variation and relative size difference of the images below has been compromised a little more.

5D2, 5D3, 40D, D800

[Mt Spokane Photography]

Its pretty apparent from the comments that intrepreting what you are showing is not being understood the same way by users.  Thats why signal to noise ratios are a good way to help people understand.
 
 
Here is how I view it:
 
The luminance (brightness of the noise) is what causes you see the noise in a image, and, as the author notes, its greatly compressed in the examples below. 
 
At the same exposure levels with a 40D (And I loved all 5 of mine), the noise image below is brighter while the noise of a 5D MK III is darker and almost invisible.  Even a bigger difference before compressing them for publication here.
 
Its the brightness or luminance of the noise with respect to the image signal (S/N ratio) that determines how much noise shows in your image.
 
Thus, in the darker images here, the noise will be less in your image and in the lighter or brighter images, noise will be more easily seen.
 
The retention of detail is related as well, but is very important.

[Aglet]

Its pretty apparent from the comments that intrepreting what you are showing is not being understood the same way by users.  Thats why signal to noise ratios are a good way to help people understand.

- clip -

Its the brightness or luminance of the noise with respect to the image signal (S/N ratio) that determines how much noise shows in your image.
 
- clip-

Nicely and succinctly re-stated, thank you.
This is what I'll be trying to convey to a wider audience on my (as yet incomplete) techblog on this topic by showing some real-world samples.
http://a2bart.com/tech/darknoise.htm

however, SNR can be looked at in imaging in a couple different ways and that's also important:

1. average signal to average noise ratio - not good as it obscures the impact large value (bright) noise can have
This seems to be a method a many review sites use (e.g. DxOmark print measurement) and that can lead to some skewed results and unhappy buyers of noisy cameras but this is also an effect of scaling images (downwards) for printing so it's not without some merit, especially when camera resolution is quite high compared to the final print size. I.E. 20MP image file printed to a 7 or 8MP print file.

2. signal to peak noise ratio - much better representation as small bright noise is more accurately weighted against the (image) signal level and it's something we perceive very readily as our eyes are contrast detection devices.  This is even more obvious to us if there is a pattern to it, even just a simple line will become quite obvious.  This is most apparent when "pixel-peeping" at full size on screen but can also be a factor when printing very large prints where individual pixel level noise can be seen on the print at close distances.  or when using noisy lower resolution image files to make moderate size prints.


I've actually got some really good examples of this from real world shots to show, just no time to prep them yet.

[zwoop]

Did you account for ISO differences between cameras?

For example, ISO 200 on the 5D2 is about ISO 143. The 50D's ISO 100 is about ISO 157. DxoMark has all the numbers.

http://www.adorama.com/alc/article/11227

[marekjoz]

Aglet -thanks for this wonderful job.
I'm a bit surprised somewhere there:
1. 7d looks better than 5d2, what is difficult to be confirmed by me as I use both.
2. I'm surprised with g1x which looks awfull in comparison to g12.

Because of question No 1: have you assured the same conditions during tests? As conditions for Darks I can't imagine anything else as same temperature (also not so, that camera has shot before 100 photos and after that you took a sample dark frame). Were the baterries fully charged? Noise reduction of course Off?
What was the exposure time? Have you tried different exposure times and compare it also in this dimension?

I'm asking just because of observation No 1 - If my experiences from using both show something different,  then how can I trust and interprete the rest?

[TrumpetPower!]

First, pushing shadows by four stops is about as meaningless as it gets. If you were to pull the highlights by an equivalent amount, you'd be attempting to add eight stops of dynamic range to the camera, basically doubling your DR. If you're trying to do that in the real world, either you know full well that the results are going to be crap or you're merging multiple exposures with HDR.

But even if you're only doing this to make the quality of the noise more visible, it's still a meaningless test because you're comparing 100% crops. A 100% crop of a 50 megapickle image can look much, much worse than a 100% crop of a 10 megapickle image and yet the 50 megapickle image might make a print that blows away the 10 megapickle one.

If you're serious about this, what you want is to pick a representative final print size -- 13" x 19" would be typical of the largest likely print size owners of those cameras might have in common -- and then scale, up or down as needs be, all the images to 100 ppi at that size. For this example, that would be 1300 x 1950 (to preserve the aspect ratio). You can then take a 100% crop of the 1300 x 1950 resized image for your comparison.

Why only 100 ppi instead of the 300 ppi you'd be printing at? Well, 300 ppi would make sense if you were going to print these, but you and everybody else will be making the comparisons on screen and not on paper.

Of course, the real way to do this kind of comparison would be with real-world images that get put through the full post-production workflow all the way to final prints, and then compare those prints...but nobody seems to do that sort of thing. I'm guessing it's because those who make prints don't waste their time with these sorts of tests, and those who waste their time with these sorts of tests don't make prints.

Cheers,

b&

[Marsu42]

Aglet -thanks for this wonderful job.  I'm a bit surprised somewhere there: 7d looks better than 5d2, what is difficult to be confirmed by me as I use both.

First, thanks for the effort to Aglet! My 2 cents critique:

* Comparing Canon to Sony's sensors (Nikon, Pentax): It is indeed the question how valid such an extreme underexposure and successive +ev correction is, or Canon would be broke by tomorrow. My idea: maybe another test with a dark grey background and moderate underexposure and post-processing +ev correction would put this into perspective? This is because that somewhat clever post-processing (software) will never try to raise black parts as far as you did.

* To me, the thing that kills my shots is banding. I guess there is no scientific number to measure this? If not, downsized shots would indeed offer a better basis for comparison since they'd show how much noise *pattern* is left.

* But most of all, I am stunned how much worse the 7d looks in this artificial test in comparison to the 60d. I have heard that the 7d is more prone to banding, but I'd never dared to mention it here because I'd get stomped. But looking at your 7d/60d pictures... am I missing something? They do use the same sensor, don't they? Is this banding due to some other processing in the 7d's dual digic cpus?

[Axilrod]

It's insane how much less noise the Nikons produce at lower ISO's, and wow the D4 handles it very well.  I am also surprised how well the 7D does against the 5DII and 5DIII, in my experience the 7D is always noisier and I hate shooting over ISO800 with it (for video anyways).  It could be that the greenish noise from the 7D is just more visible on video, not sure though.

[LetTheRightLensIn]

I managed to get my mitts on another NINE cameras this weekend and have posted the results along with the previous 15 on one page. A big wide monitor is ideal to look at the whole picture.

Now, not only are there the usual suspects from Canon; 5D2, 5D3, 7D, 60D, 50D, 40D, etc. The G1 X has been added to the PowerShot group.

There are now the new Nikon D4 (seriously dark), and D800 along with previous D5100 and D90.

Also new to the comparison are Pentax medium format 645D, APS-C sensored K5, K01 mirrorless, and even the tiny Pentax Q.
Also Sony A900 full frame SLR and Panasonic G1X four-thirds class camera.

All are pushed +4EV using my usual procedure.

http://a2bart.com/tech/tech.htm


I handled a really neat little Olympus OM D but only a pre-production model so I couldn't get files from it.  It did fit the hand nicely and featured plenty of dedicated controls on a squarish body with a bit of a retro look.  The grip and extra battery (separate units) made nice additions to the basic body.

So, I've had the chance to play with a LOT of cameras this weekend, learned a few things and shook up my imaging world a little more.  For instance, have you seriously looked at the Pentax K5?... What a control-freak's ultimate camera.  It may not get used like the workhorses like the 5D3 are but it has a lot of things that would appeal to the creative shooter.
Pocketable Pentax Q puts out impressive images for such a tiny little unit too and it sure is "cute."  A couple more lenses for it would be nice.

Unfortunately I forgot to get files from the new Fuji X-Pro 1. DANG.

I DID remember to order my next new imaging tool though. (for delivery sometimes later this year)

Well, I don't mean to do a whole whack of reviews here, just notify those who are interested in dark frame noise tests that there's a bunch more to look at now.
It'll still take me a while to post the whole-sensor reduced images to show larger scale noise patterns but some of them are pretty interesting and may reveal a few things going on inside some cameras for those with a serious tech interest.

yeah those exmors are just amazing at low ISO

the 5D3 does clean up deep shadows compared to 5D2 at high iso and the exmors no longer do better up there

all as expected (well D4 looks better than expected) but it's interesting to see it visually and wow you compared to many, many that have never been seen this way before.

you can see the nasty 7D vertical banding, it had a special form of it that none of the other canon cameras had, it also varies quite a bit from copy to copy of the 7D and across the frame, with some copies have noticeably more than others overall and most having it vary in degree across the frame

[LetTheRightLensIn]

I managed to get my mitts on another NINE cameras this weekend and have posted the results along with the previous 15 on one page. A big wide monitor is ideal to look at the whole picture.

Now, not only are there the usual suspects from Canon; 5D2, 5D3, 7D, 60D, 50D, 40D, etc. The G1 X has been added to the PowerShot group.

There are now the new Nikon D4 (seriously dark), and D800 along with previous D5100 and D90.

Also new to the comparison are Pentax medium format 645D, APS-C sensored K5, K01 mirrorless, and even the tiny Pentax Q.
Also Sony A900 full frame SLR and Panasonic G1X four-thirds class camera.

All are pushed +4EV using my usual procedure.

http://a2bart.com/tech/tech.htm


I handled a really neat little Olympus OM D but only a pre-production model so I couldn't get files from it.  It did fit the hand nicely and featured plenty of dedicated controls on a squarish body with a bit of a retro look.  The grip and extra battery (separate units) made nice additions to the basic body.

So, I've had the chance to play with a LOT of cameras this weekend, learned a few things and shook up my imaging world a little more.  For instance, have you seriously looked at the Pentax K5?... What a control-freak's ultimate camera.  It may not get used like the workhorses like the 5D3 are but it has a lot of things that would appeal to the creative shooter.
Pocketable Pentax Q puts out impressive images for such a tiny little unit too and it sure is "cute."  A couple more lenses for it would be nice.

Unfortunately I forgot to get files from the new Fuji X-Pro 1. DANG.

I DID remember to order my next new imaging tool though. (for delivery sometimes later this year)

Well, I don't mean to do a whole whack of reviews here, just notify those who are interested in dark frame noise tests that there's a bunch more to look at now.
It'll still take me a while to post the whole-sensor reduced images to show larger scale noise patterns but some of them are pretty interesting and may reveal a few things going on inside some cameras for those with a serious tech interest.

yeah it does fit, the 40D always seemed to pushable at low ISO and the 50D not so much at all

the 40D/1D3/1Ds3 gen had some of the best low iso pushability and the 50D/5D2 gen some of the least, it was always said

the 5D3 is a bit sad though for a latest generation, it loses badly to the D4 at high iso and low iso (and the D4 doesn't even use exmor technology, so even without patent busting, Canon should be able to do a lot better than they are for low ISO)

the 60D, per 100% crop, actually does a touch better than the 5D3, even, it seems

the D800 does start messing up the shadow depths ISO3200+ compared to 5D3 though

i have a weird feeling the 1DX will test closer to the D4 than the D800 and Canon simply re-used cheaper tech for the 5D3, we will see, a real shame, since a bit better sensor and it would be a true beast of camera, but if they cheaped out on the sensor and charged more than competition, not so winning....

(of course these are not normalized for ISO rating differences, which make the 5D2/50D look even worse in comparison for instance, nor for MP count, which then bring the 5D2/50D back to looking better, accounting for both at once would need to be done to truly compare them all)

[LetTheRightLensIn]

It's insane how much less noise the Nikons produce at lower ISO's, and wow the D4 handles it very well.  I am also surprised how well the 7D does against the 5DII and 5DIII, in my experience the 7D is always noisier and I hate shooting over ISO800 with it (for video anyways).  It could be that the greenish noise from the 7D is just more visible on video, not sure though.

Don't forget that mid-tone SNR and near black noise performance are not the same thing.

The 5D3 collects light over 2.56x more surface than the 7D so, per frame, it should certainly have less noise, outside of the very deepest, darkest parts of the image than the 7D.

The D4 looks pretty remarkable doing so well at low iso and at high iso for shadow noise.

The 5D3 does look 1.5-2 stops better for near black shadow noise than the 5D2 (lighter shadows and up the difference should be much less though, hopefully 2/3 of a stop but perhaps only 1/3).

One strange thing though is I havent seen numerical data hinting at the D4 being so many stops better DR at high ISO or the 5D3 being that many stops DR better there either. Something seems odd even considering ISO gains are not normalized and MP counts are not normalized.

What software was used to make these? Pre-raw processing software or just ACR which might do different things under the scenes for each?

Anyway interesting, never seen such an extensive comparison of this type before.

[Hill Benson]

Very interesting read. I'm sure it took a bit of time to get all that together. Thanks for sharing.