zwoop said:
Did you account for ISO differences between cameras?
For example, ISO 200 on the 5D2 is about ISO 143. The 50D's ISO 100 is about ISO 157. DxoMark has all the numbers.
http://www.adorama.com/alc/article/11227
HI, Thanks for asking.
I gave it only slight consideration as it adds a lot of work to normalize all those test shots using their tested actual ISO. And, since this isn't my day-job
My reasoning is that pretty much all the decent cameras out there have an actual ISO that should be within 1/2 stop of rated ISO you set them to so the worst case would be comparing one that's a half stop low to another that's a half stop high; and the latter rarely ever happens from what I've seen on DxOmark. My intent here is to show the noise structures of various cameras across their ISO range in a way they can be loosely compared.
More importantly, it is the structure of the noise distribution on the image, not just the total amount of it, that is key to how well your images may hold up in post before the dreaded banding type noise structures put a halt to your intentions.
The comparisons I've put up so far will at least allow people to see for themselves the type of noise these camera sensors are generating on a small scale relative to the whole image. When I get the whole-sensor noise map up it's another story. Larger noise structures may then show up, as they do on the 40D sample near the beginning of this thread where you can see lots of strong horizontal lines.
====
marekjoz said:
Aglet -thanks for this wonderful job.
I'm a bit surprised somewhere there:
1. 7d looks better than 5d2, what is difficult to be confirmed by me as I use both.
2. I'm surprised with g1x which looks awfull in comparison to g12.
Because of question No 1: have you assured the same conditions during tests? As conditions for Darks I can't imagine anything else as same temperature (also not so, that camera has shot before 100 photos and after that you took a sample dark frame). Were the baterries fully charged? Noise reduction of course Off?
What was the exposure time? Have you tried different exposure times and compare it also in this dimension?
I'm asking just because of observation No 1 - If my experiences from using both show something different, then how can I trust and interprete the rest?
Glad you find this info helpful or at least interesting.
Re: 1. IMO, I don't think the 7D looks better than the 5d2 in this comparison. They have similar levels of average noise at similar ISO settings but the 7D's noise is more coarse and has more obvious vertical banding at low ISO settings. Vertical banding on the 5d2 is more subtle and cleans up a little better in post with some software.
Re: 2. Yes, G1X is a very different beast with a Canon CMOS sensor in it vs the previous G12 which, I think, uses a Sony-made sensor. I was actually surprised the G12 was noticeably less noisy than the G11 despite the G12 having more advanced video features on it.
As for testing conditions, all the above cameras were tested at ~21C ambient temperature and were only operating for a minute or so before starting the test shots to turn off any in-camera NR or other toning. Batteries actually were all right near fully charged for the above cameras. The set of cameras I added this weekend had battery charge at various levels (beyond my control) and temperatures may have been a little higher as some of them had been operating occasionally prior to my testing.
Exposure times have all been done at 1/200s. An arbitrary choice on my part but picked to be representative of a typical daylight landscape exposure speed; fast enough for the sensor not to exhibit any hot pixels that would add another dimension to the testing.
i have also done some long exposure tests and long exposure with in-camera dark-frame subtraction NR applied. The results are surprising!
i will get around to posting them sometime too.
===
TrumpetPower! said:
First, pushing shadows by four stops is about as meaningless as it gets. If you were to pull the highlights by an equivalent amount, you'd be attempting to add eight stops of dynamic range to the camera, basically doubling your DR. If you're trying to do that in the real world, either you know full well that the results are going to be crap or you're merging multiple exposures with HDR.
But even if you're only doing this to make the quality of the noise more visible, it's still a meaningless test because you're comparing 100% crops. A 100% crop of a 50 megapickle image can look much, much worse than a 100% crop of a 10 megapickle image and yet the 50 megapickle image might make a print that blows away the 10 megapickle one.
If you're serious about this, what you want is to pick a representative final print size -- 13" x 19" would be typical of the largest likely print size owners of those cameras might have in common -- and then scale, up or down as needs be, all the images to 100 ppi at that size. For this example, that would be 1300 x 1950 (to preserve the aspect ratio). You can then take a 100% crop of the 1300 x 1950 resized image for your comparison.
Why only 100 ppi instead of the 300 ppi you'd be printing at? Well, 300 ppi would make sense if you were going to print these, but you and everybody else will be making the comparisons on screen and not on paper.
Of course, the real way to do this kind of comparison would be with real-world images that get put through the full post-production workflow all the way to final prints, and then compare those prints...but nobody seems to do that sort of thing. I'm guessing it's because those who make prints don't waste their time with these sorts of tests, and those who waste their time with these sorts of tests don't make prints.
Cheers,
b&
Thanks for your feedback but I have to disagree with you on some of your points. Of course, if you don't know the background to this whole saga I can understand your point of view.
First, it's not meaningless at all, unless you don't do any serious adjustments in post, as some of us do. .. And not necessarily as a "correction" for something missed in taking the shot but as a way to help map the maximum dynamic range of our cameras to the very small dynamic range of a print or even a digital display. You can't multi-shot HDR moving subjects worth a fly-spec, neither should you have to with 14bits worth of camera raw data.
100% crops are useful just to see the small scale noise structure, like the fine banding of the 7D and 5D2 or some of the G-series cameras.
If I wasn't serious about this I wouldn't waste more of my time on it. It's my gift to you, free information on a topic that's so far been poorly addressed and inadequately covered, leaving many confused or uninformed. I'm presenting it in a way that will hopefully help more people understand the limitations of their gear so they can work around it and be more aware when they can encounter problems between shooting and processing.
I was ignorant of my camera's shortcomings and overly relied on it, being a bit of a fanboy at the time. Then, being unable to process some of my real-world shots for large scale printing, because the banding noise was an issue, is what started this whole project. However, even little 18x12 inch "snapshots" can show some banding noise structures under some conditions.
I'm tempted to say that anyone who hasn't run into this sort of problem is either shooting with some better cameras or hasn't yet had to learn how to wring the utmost from the equipment they have.
... But i won't.
=====
Marsu42 said:
* Comparing Canon to Sony's sensors (Nikon, Pentax): It is indeed the question how valid such an extreme underexposure and successive +ev correction is, or Canon would be broke by tomorrow. My idea: maybe another test with a dark grey background and moderate underexposure and post-processing +ev correction would put this into perspective? This is because that somewhat clever post-processing (software) will never try to raise black parts as far as you did.
Perfectly valid and that is one of the things I'm trying to devise a good in-studio test for because the black levels and the shadow levels can behave differently. The tests I've done and displayed certainly do not tell the whole story. But they are a good prolog for generating more awareness that noise structure issues are of some concern under certain conditions.
Marsu42 said:
* To me, the thing that kills my shots is banding. I guess there is no scientific number to measure this? If not, downsized shots would indeed offer a better basis for comparison since they'd show how much noise *pattern* is left.
YUP. That's it and I wish there was some measurement of this. I haven't found any yet so decided to try cover this topic.
I don't often downsize my shots so for me, pixel-level evaluation is where it's at.
If you're printing smaller then some of these things do get averaged out a bit.
Some cameras, with strong but fine-grained noise and no obvious banding, will not have major issues in post as NR can often mitigate the noise without hurting the actual image too much. Another camera can have weak levels of noise but a regular banding pattern to it may make it far less amenable to being pushed in post. The comparisons I've put up so far will at least allow people to see for themselves the type of noise their camera sensors are generating on a small scale. When I get the whole-sensor noise map up it's another story. Larger noise structures may then show up, as they do on the 40D sample near the beginning of this thread.
Marsu42 said:
* But most of all, I am stunned how much worse the 7d looks in this artificial test in comparison to the 60d. I have heard that the 7d is more prone to banding, but I'd never dared to mention it here because I'd get stomped. But looking at your 7d/60d pictures... am I missing something? They do use the same sensor, don't they? Is this banding due to some other processing in the 7d's dual digic cpus?
7D uses a "dual readout" circuit for faster shooting speeds. This may be the problem, with each signal path taking alternate 8-pixel wide swipes of data?.. If they're not perfectly matched, then banding errors show up in the final file dark areas. Fine vertical banding still exists (iso200, 400) on the 60D but it's MUCH more subtle, barely perceptable for the most part. The 5D2 is a bit worse than the 60D.
I've tested a second 7D and 5d2, they were nearly identical with the first samples. I suspect there may be some 7Ds that are cleaner, some that may be worse. if it weren't for the 8-pixel banding, I'd be pretty happy with mine!
=====
LetTheRightLensIn said:
all as expected (well D4 looks better than expected) but it's interesting to see it visually
it is impressive - I hope there wasn't some hidden NR feature that I didn't find and turn off! The thing is almost TOO good to believe.
LetTheRightLensIn said:
you can see the nasty 7D vertical banding, it had a special form of it that none of the other canon cameras had, it also varies quite a bit from copy to copy of the 7D and across the frame, with some copies have noticeably more than others overall and most having it vary in degree across the frame
I've only looked at mine and one other and they both exhibit virtually identical noise behavior across the frame but I'd not be surprised there's variations, given the design.
LetTheRightLensIn said:
.. i have a weird feeling the 1DX will test closer to the D4..
I certainly hope so. I was able to play with a pre-production one a little. menu is very similar to the 5d3, overall handling and feel was pretty sweet with decent ergonomics for the 10 minutes I played with it. If it wasn't for that sealed memory card door!...
LetTheRightLensIn said:
One strange thing though is I havent seen numerical data hinting at the D4 being so many stops better DR at high ISO or the 5D3 being that many stops DR better there either. Something seems odd even considering ISO gains are not normalized and MP counts are not normalized.
What software was used to make these? Pre-raw processing software or just ACR which might do different things under the scenes for each?
Anyway interesting, never seen such an extensive comparison of this type before.
All processing done in ACR, so yes, could be some differences in the way it treats various camera files that user controls can't access. Best tool i've got for the job at the moment tho.
OTOH, the K5, D4, D800 and even the sub $700 D5100 all start off about the same in this race and the $ value starts to sort itself out as you move up the ISO scale.
This is still more of a noise measurement than a signal to noise measurement at this time.
Thus the various suggestions of doing this again at a low exposure value (not = to black) will bring a different context to put this in which will be more representative of real-world images and also may show some different behaviors from the cameras.
It's on the list of things to do.
