Pixel density, resolution, and diffraction in cameras like the 7D II

[RGF]

This leads me to a question that I have not received a satisfactory answer yet.

Consider an exposure at ISO 800, why is it that we can better results by setting the ISO to 800 (amplification within the camera via electronics -analog ?) versus taking the same picture at ISO 100 and adjusting exposure in the computer. Of course I am talking about a raw capture.

In both case the amount of light hitting the sensor will be the same, so the signal and S/N will be the same(?), but amplifying the signal in the camera via electronics seems to give a cleaning image

Thanks

I believe that this differs for so-called "ISO-less" cameras and... "non-ISO-less" (sic) cameras. Canon generally belongs to the latter cathegory.

Imagine a pipeline consisting of:
(noise injected)->(amplifier)->(noise injected)->(amplifier)

If the second injection of noise is significant, then you gain SNR by employing the first amplifier. If the first noise source is dominant, then it does not matter which amplifier you use.


The high-DR@lowISO sensors used by Sony/Nikon seems to give similar quality if you do the gain in a raw editor as if you do it in-camera. There are still disadvantages to this method, though. You cannot use the auto-exposure, in-camera preview is useless, and the histogram is hard to interpret. You gain better highlights (DR) in low-light images, though.

-h

Thanks. I think I understand most of what you are saying. HOwever the amplication via the computer is should not introduce any noise. The A to D is reduced from 12 (or 14) bits to 9 (or 11) bits for a 3 stop gain. Shadows may go from 6 bit to 3 bit. Not noise, but posterization?

 

[jrista]

I am trying to correlate the resulting image from a DSLR exposed at Rayleigh to how well a viewer of that image at it's native print size could resolve detail at an appropriate viewing distance, hence the reference to vision. In and of itself, Rayleigh is not a constant or anything like that. The reason MTF 80, 50, 10 (really 9%, Rayleigh) and 0 (or really just barely above 0%, Dawe's) are used is that they correlate to specific aspects of human perception regarding a print of said image. MTF 50 is generally referred to as the best measure of resolution that produces output we consider well resolved...sharp...good contrast and high acutance. MTF10 would be the limit of useful resolution, and does not directly correlate with sharpness or IQ...simply the finest level of detail resolvable that each pixel could be differentiated (excluding any effects image noise may have, which can greatly impact the viability of MTF10, another reason it is not particluarly useful for measuring photographic resolution.)

But deconvolution can shift MTF curves, can it not? At MTF0, it is really hard to see how any future technology might dig up any details (a hypothetic image scaler might make good details based on lower resolution images + good models of the stuff that is in the picture, but I think that is besides the point here).

Deconvolution can shift MTF curves, however for deconv algorithms to be most effective, as they would need to be at Rayleigh, and even more so Dawes, you need to know something about the nature of the diffraction you are working with. Diffraction is the convolution, and for a given point light source at Dawes, you would need to know the nature of it, the PSF, to properly deconvolve it. The kind of deconv algorithms we have today, such as denoise and debanding and the like, are useful up to a degree. We would need some very, very advanced algorithms to deconvolve complex scenes full of near-infinite discrete point light sources at MTF0. I can see deconvolution being useful for extracting and enhancing detail at MTF10...I think we can do that today.

That said, what we can do with software to an image once it is captured was kind of beyond the scope of my original point...which was really simply to prove that higher resolution sensors really do offer benefits over lower resolution sensors, even in diffraction-limited scenarios.

The fundamental question (to my mind, at least) is "given the system MTF/noise behaviour, after good/optimal deconvolution and noise reduction, how perceptually accurate/pleasing is the end-result?". Of course, my question is a lot vaguer and harder to figure than yours.

I figure that in 10 years, deconvolution will be a lot better (and faster) than today. This has a (small) influence on my actions today, as my raw files are stored long-term.

Sure, I totally agree! I've seen some amazing things done using deconvolution research, and I'm pretty excited about these increasingly advanced algorithms finding their way into commercial software. For one, I really can't wait until high quality debanding finds its way into Lightroom. People complain a lot about the DR of Canon sensors...however Canon sensors still pack in the deep shadow pixels, full of detail, underneath all that banding. Remove the banding, and you can recover completely usable deep shadows and greatly expand your DR. I've never seen it get as good as what an Exmor offers out of the box, but at least a stop, stop and a half beyond the 10-11 stops we get natively.

We can certainly look forward to improved digital signal processing and deconvolution. That again was a bit beyond the scope of the original points I was trying to make, hence the lack of any original discussion involving them.

 

[Don Haines]

The benefit of a high density APS-C really only comes into play when you can't afford that $13,000 600mm lens, meaning even if you had the 5D III, you could still only use that 400mm lens. Your in a focal-length limited scenario now. It is in these situations, where you have both a high density APS-C and a lower density FF body, that something like the 18mp 7D or a 24mp 7D II really shine. Even though their pixels aren't as good as the 5D III (assuming there isn't some radical new technology that Canon brings to the table with the 7D II), you can get more of them on the subject. You don't need to crop as much on the high density APS-C as with the lower density FF. On a size-normal basis, the noise of the APS-C should be similar to the FF, as the FF would be cropped more (by a factor of 1.6x), so the noise difference can be greatly reduced or eliminated by scaling down.

Well said! Might I add that even if one could afford the $13,000 600mm lens, for many of us who backpack it's just too large to bring along on a multi-day trek through the mountains. Sometimes bigger is not better.

 

[jrista]

Can jrista take account of the heavy AA filters on Canon sensors in his calcs or have I missed something?

Well first, as I've mentioned in the past, I believe the idea that Canon uses overly strong AA filters is a bit overblown. I thought my 7D had an aggressive AA filter until I first put on the EF 300mm f/2.8 L II in August 2012. I'd been using a 16-35 L II and the 100-400 L. Both are "good" lenses, but neither is a truly "great" lens. Both don't seem to have enough resolving power and/or contrast to really do the 7D justice. They get really close, but often enough they fall just a little short, which produces that well-known "soft" look to 7D images.

With the F 300/2.8 II, 500/4 II, and 600/4 II, with and without 1.4x and 2x TCs, the IQ of the 7D is stellar. I've never seen any of the classic softness that I did with my other lenses. Based on the IQ from using Canons better lenses, I do NOT believe they have aggressive AA filters...I think they actually have AA filters that are just right!

On pentaxforums you will see comparisons between the K5-II and the filterless K5-IIs. Both have same 16Mp sensors but the -IIs is a whole lot sharper and is claimed to be equivalent to a filtered 24Mp camera.

Does this matter in the real world? Well yes maybe if you want to use 100pc crops.

Will Canon introduce something like the D7100 in their well trailed up coming 7D2, 70D, 700D series?

Somehow doubt it.

Excellent stuff jrista - thanks for posting.

I'm curious about the difference with the Pentax K5 II. If there is that much of a difference, I'd presume that the AA filter WAS aggressive. From what I have heard, the D800 and D800E, when you use a good lens, does NOT exhibit that much of a difference. In many reviews I've read, the differences were sometimes barely perceptible, with the added cost on the D800E that if you shoot anything with repeating patterns, you can end up with aliasing and moire. There is definitely some improvement to shooting without an AA filter, but I am not sure it is really all it is cracked up to be.

Generally speaking, I would blame the lens for any general softness unless it is definitively proven to outresolve the sensor. For sensors with the densities they have today, lenses are generally only capable of outresolving the sensor in a fairly narrow band of aperture settings...from around f/3.5 to f/8 for FF sensors, and f/3.5 to f/5.6 for APS-C sensors. The higher the density of the sensor, the narrower the range....a 24mp APS-C can probably only be outresolved at around f/4, unless the lens is more diffraction-limited at wider apertures. Wider than f/3.5 in the majority of cases, optical aberrations cause softening, in many cases much more than you experience from diffraction even at f/22.

That said, so long as you pair a high quality lens with a high density sensor on a Canon camera, or for that matter a Nikon camera, I do not believe the AA filter will ever be a serious problem. When it comes to other brands, I don't really know enough. In the case of the K5 II, it really sounds more like the AA version DOES have an aggressive filter, which is why there is a large difference between the AA and non-AA versions.

On pentaxforums you will see comparisons between the K5-II and the filterless K5-IIs. Both have same 16Mp sensors but the -IIs is a whole lot sharper and is claimed to be equivalent to a filtered 24Mp camera.

If you check out luminous-landscape.com there is a nice thread comparing optimally sharpened D800 vs optimally sharpened D800E. I believe the conclusion is that for low-noise situations, the performance is virtually identical.

Aye, this is what I've heard as well. There is a small improvement with the D800E in the right circumstances, but overall it does not seem to be as great as it otherwise sounds on paper. Given the IQ I can get out of the 7D, which does have an AA filter, with top-shelf lenses...I really do not believe it has an aggressive AA filter, and I am quite thankful that the AA filter is there. Without it, I'd never be able to photograph birds...their feathers are moire hell!

 

[3kramd5]

Such a sensor would really be pushing the limits, as well, and probably wouldn't even be physically possible. The pixel pitch of such a sensor would be around 723 nanometers (0.723µm)!

Nokia got halfway there (1.4 micron) with a camera phone...

 

[jrista]

Some interesting points here, though not all of them are 100% based on optical reality (which has a measurable match in optical theory to 99+% most of the time, as long as you stay within Fraunhofer and quantum limits)

Rayleigh is indeed the limit where MTF has sunk to 7% (or 13%, depending if your target is sine or sharp-edge, OTF) - a point where it is very hard to recover any detail by sharpening if your image contains any higher levels of noise. It's hard even at low levels of noise. And Rayleigh is defined as: "When the peak of one point maxima coincides with the first Airy disk null on the neighboring point".

Consider again what this means.
You have two points, at X distance p-p. The distance you're interested in finding is where they begin to merge enough to totally mask out the void in between. Rayleigh distance gives "barely discernible differentiation, you can JUST make out that there's two spots, not one wide spot.

But this does not make Rayleigh the p-p distance of pixels needed to register that resolution, to register the void in the first place you have to have one row of pixels between the two points. That means that Rayleigh is a line-pair distance, not a line distance. If Rayleigh is 1mm and the sensor 10mm, you need to have 20 pixels to resolve it.

You've hit it on the head...measuring resolution from a digital sensor at Rayleigh is very difficult because of noise. The contrast level is so low that, when the image is combined with noise (both photon shot and read), there is really no good way to discern whether the difference between two pixels is caused by differences in the detail resolved from the scene, or differences caused by noise.

There is probably a better "sweet spot" MTF, lower than MTF 50 but higher than MTF @ Rayleigh, that would give us a better idea of how well digital image sensors can resolve detail. Given the complexities involved with using MTF @ Rayleigh, and the ubiquitous acceptance of MTF 50 as the best measure of resolution from a human perception standpoint, I prefer to use MTF 50.

 

[jrista]

"Rayleigh distance gives "barely discernible differentiation, you can JUST make out that there's two spots, not one wide spot."

I guess this is the point of discussion. Like I said earlier, if you have some knowledge of the scene (such as knowing that it is dirac-like stars on a dark sky), or if you have knowledge of the PSF/low noise/access to good deconvolution, you can challenge this limit.

I have no issues with the Rayleigh criterion being a practically important indicator of "diminishing returns". I do have an issue with people claiming that it is an absolute brickwall (nature seems to dislike brickwall filters - perhaps because it assumes acausality?).

From a theoretic perspective it would be interesting to know if there are true, fundamental limits to the information passed onto the sensor (a PSF will to some degree "scramble" the information, making it hard to sort out. That is not the same as removing it altogether). I have seen some hints of such a limit, but I never had much optical physics back in University, and I am too lazy to read up on the theory myself. There were a discussion on dpreview where they talked about a few hundred megapixels/ gigapixel for an FF sensor before the blue sensels could not receive any more spatial information.

At the very least, I assume that we move into quantum trouble sooner or later. When a finite number of photons hits a sensor, there is only so much information to record. If you cannot simultaneously record the precise position and energy of a photon, then that is a limit. I assume that as a sensel approach the wavelength of light, nastyness happens. That is one more area of physics that I do not master.

-h

I don't think I've claimed Rayleigh is a "brick wall". I'd call Dawe's the brick wall, as anything less than that and you have two unresolved points of light. The problem with Rayleigh, at least in the context of spatial resolution in the photographic context...is that detail becomes nearly inseparably mired with noise. At very low levels of contrast, even assuming you have extremely intelligent and effective deconvolution, detail at MTF 10 could never really be "certain"....is it detail...or is it noise? Even low levels of noise can have much higher contrast than detail at MTF 10. Dawe's Limit is the brick wall, Rayleigh is the effective limit of resolution for all practical intents, and leave a fairly significant degree of uncertainty in discussions like this.

MTF50 is widely accepted as being a moderately lower contrast level where detail is acceptably perceivable by the human eye for a print at native resolution. In the film days, the perception of a viewer was evaluated from contact prints, so what the film resolved is what the viewer could see. Given the broad use and recognition of MTF 50, its what I use. Ultimately, it wouldn't really matter of I used MTF 50, MTF 10, or MTF 0...the math will work out roughly the same either way, and the relative benefits of a 24mp sensor over an 18mp sensor will still exist. The MTF is really just to provide a consistent frame of reference, nothing more.

 

[jrista]

This leads me to a question that I have not received a satisfactory answer yet.

Consider an exposure at ISO 800, why is it that we can better results by setting the ISO to 800 (amplification within the camera via electronics -analog ?) versus taking the same picture at ISO 100 and adjusting exposure in the computer. Of course I am talking about a raw capture.

In both case the amount of light hitting the sensor will be the same, so the signal and S/N will be the same(?), but amplifying the signal in the camera via electronics seems to give a cleaning image

Thanks

I believe that this differs for so-called "ISO-less" cameras and... "non-ISO-less" (sic) cameras. Canon generally belongs to the latter cathegory.

Imagine a pipeline consisting of:
(noise injected)->(amplifier)->(noise injected)->(amplifier)

If the second injection of noise is significant, then you gain SNR by employing the first amplifier. If the first noise source is dominant, then it does not matter which amplifier you use.


The high-DR@lowISO sensors used by Sony/Nikon seems to give similar quality if you do the gain in a raw editor as if you do it in-camera. There are still disadvantages to this method, though. You cannot use the auto-exposure, in-camera preview is useless, and the histogram is hard to interpret. You gain better highlights (DR) in low-light images, though.

-h

Thanks. I think I understand most of what you are saying. HOwever the amplication via the computer is should not introduce any noise. The A to D is reduced from 12 (or 14) bits to 9 (or 11) bits for a 3 stop gain. Shadows may go from 6 bit to 3 bit. Not noise, but posterization?

"Amplification" via software does not introduce noise...however it can enhance the noise present, because at that point, assuming noise exists in the digital signal, it is "baked in". When it comes to Exmor (Sony/Nikon high-DR sensor), the level of noise is extremely low, so pushing exposure around in post is "amplifying" pixels that have FAR less noise than the competition.

The Exmor sensor could be called ISO-LESS because it is primarily a DIGITAL pipeline.

In most sensors, when a pixel is read, analog CDS is applied, analog per-pixel amplification is applies, the columns of a row are read out, the signal is often sent off the sensor die via bus, a downstream analog amplifier may be applied, and the pixels are finally converted to digital by a high frequency ADC. Along that whole pipeline there are many chances for noise to be introduced to the analog signal. Canon's sensors are like this, and some of the key sources of noise are the non-uniform response of the CDS circuits (which is the first source of banding noise), transmission of the signal along a high speed bus, downstream amplification (which amplifies all the noise in the signal prior to secondary amplification...which only occurs at the highest ISO settings), and finally large bucket parallel ADC via high frequency converters (which is where the second source of banding noise comes from).

Unlike most sensors, the only analog stage in Exmor is the direct read of each pixel. Once a pixel is read, it is sent directly to an ON-DIE ADC, where pixels are converted directly to a digital form, where digital CDS is applied, where digital amplification is applied, and from which point on the entire signal remains digital. Once the signal is in a digital form, it is, for all intents and purposes, immune to contamination by analog sources of noise. Transmission along a bus, further image processing, etc. all work on bits rather than an analog signal. As such, Exmor IS effectively "ISO-less", since amplification occurs post-ADC. ISO 800 with Exmor is really the same as ISO 100 with a 3-stop exposure boost in post...there is really little difference.

 

[jrista]

The benefit of a high density APS-C really only comes into play when you can't afford that $13,000 600mm lens, meaning even if you had the 5D III, you could still only use that 400mm lens. Your in a focal-length limited scenario now. It is in these situations, where you have both a high density APS-C and a lower density FF body, that something like the 18mp 7D or a 24mp 7D II really shine. Even though their pixels aren't as good as the 5D III (assuming there isn't some radical new technology that Canon brings to the table with the 7D II), you can get more of them on the subject. You don't need to crop as much on the high density APS-C as with the lower density FF. On a size-normal basis, the noise of the APS-C should be similar to the FF, as the FF would be cropped more (by a factor of 1.6x), so the noise difference can be greatly reduced or eliminated by scaling down.

Well said! Might I add that even if one could afford the $13,000 600mm lens, for many of us who backpack it's just too large to bring along on a multi-day trek through the mountains. Sometimes bigger is not better.

That's a great point! Sometimes bigger is definitely not better...if I was hiking around Rocky Mountain National Park, I'd probably not want to bring anything larger than the 100-400mm L.

 

[jrista]

Deconvolution can shift MTF curves, however for deconv algorithms to be most effective, as they would need to be at Rayleigh, and even more so Dawes, you need to know something about the nature of the diffraction you are working with. Diffraction is the convolution, and for a given point light source at Dawes, you would need to know the nature of it, the PSF, to properly deconvolve it.

If you knew the precise PSF and had no noise, then you could in theory retrieve the unblurred "original" perfectly through linear deconvolution. In practice, it is impossible to know the precise PSF, there is noise, and the PSF might contain deep spectral zeros. This is where non-linear, blind deconvolution comes into the picture. Sadly, I don't know much about how they work, but I know some Wiener filtering, and I believe that serve as a starting-point?

There are a whole lot of starting points. People are doing amazing things with advanced deconvolution algorithms these days. Debanding. Denoising. Eliminating motion blur. Recovering detail from a completely defocused image. The list of what we can do with deconvolution algorithms, particularly in the wavelet space, is long and growing. Whether it will help us really extract more resolution at contrast levels as low as or less than 10%, I can't say. I guess if you could denoise extremely well, and had a rough idea of the PSFs, then you could probably do some amazing things. I guess we'll see when amazing things start happening over the next decade.

The kind of deconv algorithms we have today, such as denoise and debanding and the like, are useful up to a degree. We would need some very, very advanced algorithms to deconvolve complex scenes full of near-infinite discrete point light sources at MTF0. I can see deconvolution being useful for extracting and enhancing detail at MTF10...I think we can do that today.

Doing anything up against the theoretical limit tends to be increasingly hard for vanishing benefits.

Certainly.

That said, what we can do with software to an image once it is captured was kind of beyond the scope of my original point...which was really simply to prove that higher resolution sensors really do offer benefits over lower resolution sensors, even in diffraction-limited scenarios.

I think that deconvolution strengthens your point, as it means that even higher resolution sensors make some sense even when operating in the "diffraction limited" regime. If the sensor is of lower resolution, then higher spatial frequencies are abruptly cutoff (or folded into lower frequencies) and impossible to recover through deconvolution or other means.

That is a good point. It is kind of along the same lines as the argument of, when you need a deep DOF, using a very narrow aperture like f/22 or f/32 despite the softening it incurs, rather than opting for a wider aperture that won't necessarily produce the DOF you need. Correcting the uniform blurring caused by diffraction is a hell of a lot easier than correcting the non-linear blurring caused by a too-thin DOF. Deconvolution is definitely a powerful post-processing tool that can enhance the use of higher resolution sensors (among other things), and realize fine detail at low contrast levels that exist, but are not readily apparent.

 

[CarlTN]

Jrista, "+1" on you starting a new thread with this. Thanks for all the useful info!

You mention NR software like Topaz for debanding, but what software works best for the luminance noise reduction?

I mentioned before, that I noticed the luminance noise via my cousin's 5D3, such as at ISO 4000, had a very hard pebble-like grain structure that gets recorded the size of maybe 5 or 6 pixels across. The luminance slider in ACR CS5 had very little effect on it until it got above 80%, so more detail was sacrificed. With my 50D's files, the luminance grain is much smaller in size relative to the pixels, so the luminance slider has a far greater effect in its lower range.

I have practiced the art of optimizing a file in ACR before I ever even open it in Photoshop, but is it possible that this isn't always the best approach for noise reduction? I still think it is, but I'm trying to be open minded and learn new things!

 

[jrista]

I don't think I've claimed Rayleigh is a "brick wall".

I am sorry, this was a general rant, not targeted at you.

I'd call Dawe's the brick wall, as anything less than that and you have two unresolved points of light.

https://www.astronomics.com/Dawes-limit_t.aspx
"This “Dawes’ limit” (which he determined empirically simply by testing the resolving ability of many observers on white star pairs of equal magnitude 6 brightness) only applies to point sources of light (stars). Smaller separations can be resolved in extended objects, such as the planets. For example, Cassini’s Division in the rings of Saturn (0.5 arc seconds across), was discovered using a 2.5” telescope – which has a Dawes’ limit of 1.8 arc seconds!"

The problem with Rayleigh, at least in the context of spatial resolution in the photographic context...is that detail becomes nearly inseparably mired with noise. At very low levels of contrast, even assuming you have extremely intelligent and effective deconvolution, detail at MTF 10 could never really be "certain"....is it detail...or is it noise?

I guess you can never be "certain" at lower spatial frequencies either? As long as we are dealing with PDFs, it is a matter of probability? How wide is the tail of a Poisson distribution?

I see no theoretical problem at MTF10 as long as the resulting SNR is sufficient (which it, of course, usually is not).

-h

Well, I am not necessarily talking about frequencies...just contrast (which would really be amplitude rather than frequency.) It would be correct to assume that low MTF can affect amplitude regardless of frequency, however I think it has a more apparent impact at higher frequencies than lower.

 

[TheSuede]

Well, I am not necessarily talking about frequencies...just contrast (which would really be amplitude rather than frequency.) It would be correct to assume that low MTF can affect amplitude regardless of frequency, however I think it has a more apparent impact at higher frequencies than lower.

Contrast is meaningless as a metric - until you have both amplitude contrast AND frequency. This is inherently implied in MTF, as it is defined as contrast over frequency.... Contrast is just a difference in brightness. It doesn't become "detail" until the contrast is present at a high spatial frequency.

In practice (I've written and also quantified many Bayer interpolation schemes) you need at least MTF20 - a Michelson contrast of 0.2 - to get better than 50% pixel estimation accuracy when interpolating a raw image (based on Bayer of course).

Those 0.2 in contrast does by physical necessity INCLUDE noise. Even the best non-local schemes cannot accurately estimate a detail on pixel level when that detail has a contrast lower than approximately twice the average noise power of the pixel surrounds.

The only way to get past this is true oversampling, and that does not occur in normal cameras until you're at F16-F22. In that case no interpolation estimation is needed - just pure interpolation. At that point you can be certain that no detail in the projected image will be small enough to "fall in between" two pixels of the same color on the sensor.

 

[jrista]

Well, I am not necessarily talking about frequencies...just contrast (which would really be amplitude rather than frequency.) It would be correct to assume that low MTF can affect amplitude regardless of frequency, however I think it has a more apparent impact at higher frequencies than lower.

Contrast is meaningless as a metric - until you have both amplitude contrast AND frequency. This is inherently implied in MTF, as it is defined as contrast over frequency.... Contrast is just a difference in brightness. It doesn't become "detail" until the contrast is present at a high spatial frequency.

In practice (I've written and also quantified many Bayer interpolation schemes) you need at least MTF20 - a Michelson contrast of 0.2 - to get better than 50% pixel estimation accuracy when interpolating a raw image (based on Bayer of course).

Those 0.2 in contrast does by physical necessity INCLUDE noise. Even the best non-local schemes cannot accurately estimate a detail on pixel level when that detail has a contrast lower than approximately twice the average noise power of the pixel surrounds.

The only way to get past this is true oversampling, and that does not occur in normal cameras until you're at F16-F22. In that case no interpolation estimation is needed - just pure interpolation. At that point you can be certain that no detail in the projected image will be small enough to "fall in between" two pixels of the same color on the sensor.

I don't think we are saying different things... I agree that having a certain minimum contrast is necessary for detail at high frequencies to be discernible as detail.

I am not sure I completely follow...some of the grammar is confusing. In an attempt to clarify for other readers, I think you are saying that because of the nature of a bayer type sensor, MTF at no less than 20% is necessary to demosaic detail from a bayer sensor's RAW data such that it could actually be perceived differently than noise in the rendered image. Again...I don't disagree with that on principal, however with post-process sharpening, you CAN extract a lot of high frequency detail that is low in contrast. The moon is a superb example of this, where detail most certainly exists at contrast levels below 20%, as low as Rayleigh, and possibly even lower.

The only time when it doesn't matter is at very narrow apertures, where the sensor is thoroughly outresolving the lens, and the finest resolved element of detail is larger than a pixel.

(I believe that is what The Suede is saying...)

 

[jrista]

Well, I am not necessarily talking about frequencies...just contrast (which would really be amplitude rather than frequency.) It would be correct to assume that low MTF can affect amplitude regardless of frequency, however I think it has a more apparent impact at higher frequencies than lower.

I am talking about something like this:

Increasing (spatial) frequency from left to right, increasing contrast from top to bottom. As we are talking about light and imaging, I think that amplitude/phase are cumbersome properties.

Another way to put that would be Frequency (low to high) on the X axis, and Amplitude (high to low) on the Y axis. Contrast is simply the amplitude of the frequency wave. At the bottom, the amplitude is high, and constant across the whole length of the image, while frequency increases from left to right. At the top, the amplitude is flat. At the middle, the amplitude is about 50%, while again frequency increases from left to right.

"Spatial" frequencies are exactly that...you don't have a waveform without frequency, amplitude, and phase. Technically speaking, the image above is also modulated in both frequency and amplitude, with a phase shift of zero.

If you convolve this image with a sensible PSF, you will get blurring, affecting the high frequencies the most. As convolution is a linear process, high-contrast and low-contrast parts will be affected equally. Now, if you add noise to the image, the SNR will be more affected in low-contrast than in high-contrast areas.

With the image above, you could convolve it with a sensible PSF, and deconvolve it perfectly. Noise, however, would actually affect both the low frequency parts of the image as well as the low contrast parts. The high frequency high contrast parts are actually the most resilient against noise...everything else is susceptible (which is why noise at high ISO tends to show up much more in OOF backgrounds than in a detailed subject.)

If the "ideal mathematical idea" of this image is recorded as a finite number of Poisson-distributed photons, you get some uncertainty. The uncertainty will be largest where there are a few photons representing a tiny feature, and smallest where there are many photons representing a smooth (large) feature. My point was simply that the uncertainty is there for the entire image. However unilkely, that image that seems to portray a Lion at the zoo _could_ really be of a soccer game, only shot noise altered it.

Assuming an image affected solely by Poisson-distributed photons, then theoretically, yes. However, the notion that an image of a soccer game might end up looking like a lion at the zoo would only really be probable at extremely low exposures. SNR would have to be near zero, such that the Poisson distribution of photon strikes left the majority of the sensor untouched, leaving more guesswork and less structure in figuring out what the image is. As the signal strength increases, the uncertainty shrinks, and the chances of a soccer game being misunderstood as a lion at the zoo diminishes to near-zero. Within the first couple stops of an EV, the uncertainty drops well below 1, and assuming you fully expose the sensor (i.e. ETTR) to maximize SNR, uncertainty should be well below 0.1. As uncertainty drops, the ease with which we can remove photon noise should increase.

However...noise is a disjoint factor from MTF. The two are not mutually exclusive, however they are distinct factors, and as such they can be affected independently with high precision deconvolution. You can, for example, eliminate banding noise while barely touching the spatial frequencies of the image.

 

[TheSuede]

I am talking about something like this:

Another way to put that would be Frequency (low to high) on the X axis, and Amplitude (high to low) on the Y axis. Contrast is simply the amplitude of the frequency wave.
.../cut/...

No. Using the word "amplitude" as a straight replacement for the word "contrast" (red-marked text) - is actually very misleading.

The amplitude is not equal to contrast in optics, and especially not when you're talking about visual contrast. Contrast, as normal people speak of it, is in most cases closely related to [amplitude divided by average level]. And so are MTF figures - this is not a coincidence.

An amplitude of +/-10 is a relatively large contrast if the average level is 20
-giving an absolute amplitude swing from 10 to 30 >> an MTF of 0.5
But if the average level is 100, then swing is 90-110 >> MTF is only 0.1. That's a very much lower contrast, and a lot harder to see or accurately reproduce.

Contrast is what we "see", not amplitude swing.

And no, noise in general is not generally disjointed from MTF... Patterned noise is separable from image detail in an FFT, and you can eliminate most of it without disturbing underlying material. Poisson noise or any other non-patterned noise on the other hand isn't separable, by any known algorithm. And since the FFT of Poisson is basically a Gauss bell curve, you remove Poisson noise by applying a Gaussian blur... Any attempt to reconstruct the actual underlying material will be - at worst - a wild guess, and - at best - and educated guess. The educated guess is still a guess, and the reliability of the result is highly dependent on non-local surrounds.

The Gaussian blur radius you need to apply to dampen non-patterned noise by a factor "X" is (again, not by coincidence!) almost exactly the same as the amount of downwards shift in MTF that you get.

As noise suppression algorithms get smarter and smarter, the amount of correct guesses-estimates in a certain image with a certain noise amount present will continue to increase (correlation to reality will get better and better) - but they're still guesses. But that's good enough for most commercial use. What we're doing today in commercial post-processing regarding noise reduction is mostly adapting to psycho-visuals. We find ways to make the viewer THINK that: -"Ah... That looks good, that must be right" - by finding what types of noise patterns that humans react strongly to, and then trying to avoid creating those patterns when blurring the image (all noise suppression is blurring!) and making/estimating new sharp edges.

 

[jrista]

I am talking about something like this:

Another way to put that would be Frequency (low to high) on the X axis, and Amplitude (high to low) on the Y axis. Contrast is simply the amplitude of the frequency wave.
.../cut/...

No. Using the word "amplitude" as a straight replacement for the word "contrast" (red-marked text) - is actually very misleading.

The amplitude is not equal to contrast in optics, and especially not when you're talking about visual contrast. Contrast, as normal people speak of it, is in most cases closely related to [amplitude divided by average level]. And so are MTF figures - this is not a coincidence.

An amplitude of +/-10 is a relatively large contrast if the average level is 20
-giving an absolute amplitude swing from 10 to 30 >> an MTF of 0.5
But if the average level is 100, then swing is 90-110 >> MTF is only 0.1. That's a very much lower contrast, and a lot harder to see or accurately reproduce.

Contrast is what we "see", not amplitude swing.

And no, noise in general is not generally disjointed from MTF... Patterned noise is separable from image detail in an FFT, and you can eliminate most of it without disturbing underlying material. Poisson noise or any other non-patterned noise on the other hand isn't separable, by any known algorithm. And since the FFT of Poisson is basically a Gauss bell curve, you remove Poisson noise by applying a Gaussian blur... Any attempt to reconstruct the actual underlying material will be - at worst - a wild guess, and - at best - and educated guess. The educated guess is still a guess, and the reliability of the result is highly dependent on non-local surrounds.

The Gaussian blur radius you need to apply to dampen non-patterned noise by a factor "X" is (again, not by coincidence!) almost exactly the same as the amount of downwards shift in MTF that you get.

As noise suppression algorithms get smarter and smarter, the amount of correct guesses-estimates in a certain image with a certain noise amount present will continue to increase (correlation to reality will get better and better) - but they're still guesses. But that's good enough for most commercial use. What we're doing today in commercial post-processing regarding noise reduction is mostly adapting to psycho-visuals. We find ways to make the viewer THINK that: -"Ah... That looks good, that must be right" - by finding what types of noise patterns that humans react strongly to, and then trying to avoid creating those patterns when blurring the image (all noise suppression is blurring!) and making/estimating new sharp edges.

Well, I can't speak directly to optics specifically.

I was thinking more in the context of the image itself, as recorded by the sensor. The image is a digital signal. There is more than one way to "think about" an image, and in one sense any image can be logically decomposed into discrete waves. Any row or column of pixels, block pixels, however you want to decompose it, could be treated as a Fourier series. The whole image can even be projected into a three dimensional surface shaped by a composition of waves in the X and Y axes, with amplitude defining the Z axis.

Performing such a decomposition is very complex, I won't deny that. Sure, a certain amount of guesswork is involved, and it is not perfect. Some algorithms are blind, and use multiple passes to guess the right functions for deconvolution, choosing the one that produces the best result. It is possible, however, to closely reproduce the inverse of the Poisson noise signal, apply it to the series, and largely eliminate that noise...with minimal impact to the rest of the image. Banding noise can be removed the same way. The process of doing so accurately is intense, and requires a considerable amount of computing power. And since a certain amount of guesswork IS involved, it can't be done perfectly without affecting the rest of the image at all. But it can be done fairly accurately with minimal blurring or other impact.

Assuming the image is just a digital signal, which in turn is just a composition of discrete waveforms, opens up a lot of possibilities. It would also mean that, assuming we generate a wave for just the bottom row of pixels in the sample image (the one without noise)...we have a modulated signal of high amplitude and decreasing frequency. The "contrast" of each line pair in that wave is fundamentally determined by the amplitude of the wavelet. The row half-way up the image would have half the amplitude...which leads to what we would perceive as less contrast.

Perhaps it is incorrect to say that amplitude itself IS contrast, I guess I wouldn't dispute that. A shrinking amplitude around the middle gray tone of the image as a whole does directly lead to less contrast as you move up from the bottom row of pixels to the top in that image. Amplitude divided by average level sounds like a good way to describe it then, so again, I don't disagree. I apologize for being misleading.

I'd also offer that there is contrast on multiple levels. There is the overall contrast of the image (or an area of the image), as well as "microcontrast". If we use the noisy version of the image I created, the bottom row could not be represented as a single smoothly modulated wave. It is the combination of the base waveform of increasing frequency, as well as a separate waveform that represents the noise. The noise increases contrast on a per-pixel level, without hugely affecting the contrast of the image overall.

Perhaps this is an incorrect way of thinking about real light passing through a real lens in analog form. I know far less about optics. I do believe Hjulenissen was talking about algorithms processing a digital image on a computer, in which case discussing spatial frequencies of a digital signal seemed more appropriate. And in that context, a white/black line pair's contrast is directly affected by amplitude (again, sorry for the misleading notion that amplitude IS contrast...I agree that is incorrect.)

 

[Radiating]

Canon WANTS diffraction to be a limiting factor so that they can remove the AA filter.

If you look at a sharp lens at f11 like a super telephoto and a soft lens at f/11 the sharp lens looks sharper despite being at the diffraction limit.

What 24MP does is it allows the whole system to be sharper due to a weaker AA filter. Diffraction is the best AA filter on earth, current ones degrade the image by 20% which is a lot.

 

[jrista]

Canon WANTS diffraction to be a limiting factor so that they can remove the AA filter.

If you look at a sharp lens at f11 like a super telephoto and a soft lens at f/11 the sharp lens looks sharper despite being at the diffraction limit.

What 24MP does is it allows the whole system to be sharper due to a weaker AA filter. Diffraction is the best AA filter on earth, current ones degrade the image by 20% which is a lot.

Where do you get that 20% figure? I can't say I've experienced that with anything other than the 100-400 @ 400mm f/5.6...however in that case, I presume the issue is the lens, not the AA filter...

 

[jrista]

Perhaps this is an incorrect way of thinking about real light passing through a real lens in analog form. I know far less about optics. I do believe Hjulenissen was talking about algorithms processing a digital image on a computer, in which case discussing spatial frequencies of a digital signal seemed more appropriate. And in that context, a white/black line pair's contrast is directly affected by amplitude (again, sorry for the misleading notion that amplitude IS contrast...I agree that is incorrect.)

"Light" is an electro-magnetic wave, or can at least be treated as one. Radio is also based on electro-magnetic waves. In radio, you have coherent transmitters and receivers, and properties of the waveform (amplitude, phase, frequency) can contain information, or be used to improve directional behaviour etc. In regular imaging applications, we tend to treat light in a somewhat different way. The sun (and all other illuminators except LASERs) are incoherent. Imaging sensors dont record the light "waveform" at a given spatial/temporal region, but rather records intensity within a frequency band. This treats light in a slightly more statistical manner, just like static noise on a radio. What is the frequency of filtered white noise? What is its phase? Amplitude? Such terms does not make sense, but its Spectral Power Density, its Variance does make sense.

Well, I understand the nature of light, its wave particle duality, all of that. I am just not an optical engineer, so I am not sure if there is any knowledge in that field that would give a different understanding to exactly what happens to light in the context of optical imaging. That said, you are thinking about light as a particle. I am actually not thinking about light at all...but rather the spatial frequencies of an image, or in the context of a RAW image on a computer (well past the point where physical light is involved), a digital signal.

I'm not sure if I can describe it such that you'll understand or not...but think of each pixel as a sample of a wave. Relative to its neighboring pixels, it is either lighter, darker or the same tone. If we have a black pixel next to a white pixel, the black pixel is the trough of a "wave", and the white pixel is the crest. If we have a white-black-white-black, we have two full "wavelengths" next to each other. The amplitude of a spatial wave is the difference between the average tone and its trough or crest. In the case of our white-black-white-black example, the average tone is middle gray, Spatial frequencies exist in two dimensions, along both the X and the Y axis. I'll see if I can find a way to plot one of the pixel rows as a wave.

In the image file that I attached, once printed and illuminated using a light bulb (or the sun), it is the intensity that is modulated (possibly through some nonlinear transform in the printer driver). The amount of black ink varies, and this means that more photons will be absorbed in "black" regions than in "white". The amplitude and phase properties of the resultant light is of little relevance. The frequency properties are also of little relevance as the image (and hopefully the illuminant) should be flat spectrum. If you point your camera to such a print, it will record the number of photons (again, intensity).

When it comes to the modulation of the intensity in the figure, this was probably done by a sinoid of sweeping frequency (left-right) and amplitude (up-down). The phase of the modulation does not matter much, as we are primarily interested in how the imaging system under test reduce the modulation at different spatial frequencies, and (more difficult) if this behaviour is signal-level dependent (like USM sharpening would be). If you change the phas of the modulation by 180 degrees, you would still be able to learn the same about the lense/sensor/... used to record the test image.

So, again, all that is thinking about light directly as a waveform or particle. That is entirely valid, however there are other ways of thinking about the image produced by the lens. The image itself is comprised of frequencies based on the intensity of a pixel. A grayscale image is much easier to demonstrate with than a color image, so I'll use that to demonstrate:

The image above models the bottom row of pixels from your image as a spatial frequency. I've stretched that single row of pixels to be 10 pixels tall, simply so it can be seen better. I've plotted the spatial waveform below. The concept is abstract...it is not a physical property of light. It is simply a way of modeling the oscillations inherent to the pixels of an image based on their level. This is a very simplistic example...we have the luxury of an even-toned middle gray as our "zero energy" level. Assuming the image is an 8-bit image, we have 256 levels. Levels 0-127 are negative power, levels 128-255 are positive power. The intensity of each pixel in the image oscillates between levels 0 and 255, thus producing a wave...with frequency and amplitude. Phase exists too...if we shift the whole image to the left or right, and appropriately fill in the pixels to the opposite side, we have all of the properties that represent a wave.

Noise can be modeled the same way...only as a different wave with different characteristics. The noise channel from the full sized image below is shown at the bottom of the wave model above (although it is not modeled itself...can't really do it well in photoshop.) Thought of as a Fourier series, the noise wave and the image wave are composible and decomposible facets of the image.

MTF with Noise:

MTF Plot:

Noise channel: