Patent: Subdividing High & Low Sensitivity Pixels

[MrToes]

Re: Patent: Subdividing High & Low Sensitivity Pixels

By keeping their ADC's off chip, this could actually be simpler to implement because of it. It just means more ADC circuitry and accompanying processes on the main board which is easily doable. So while the sony sensor may have a lower noise floor and therefor a greater dynamic range from on sensor ADCs. This can accomplish the same thing with a lower sensitivity pixel and 2 higher sensitivity pixels, combine them together and get the same or more dynamic range.

I think it's doable given canon knows how to split a pixel.

What ever they do, they better do it soon. Slowly Canon Glass is moving over in the bag for Nikon Glass.

I've been trying the D810, Not as good as some say it is but it does have some amazing low ISO DR! I do love the Nikon exposure on the focus spot also! Wish Canon would implement that on like 5DS R and future cameras.

Hope to see the new sensor tech soon and Canon leading the pack again!

 

[RGF]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Split pixels promises to have advantages. Wonder how soon we will see this in a camera.

 

[StudentOfLight]

Re: Patent: Subdividing High & Low Sensitivity Pixels

They (Canon) can get a lot of performance gain by simply moving the signal chain and digitization onto the sensor chip. It would be a lot more conservative (both technically and financially) to do that than it would be to move wholesale to this newly disclosed approach. I believe there is ZERO chance of this making it into a commercial product in the next 12 months. This might be in the 1DX3 or 5D5.

If you move lot of stuff on the sensor chip, it'll heat up and add noise.

If you go to finer lithography you can increase speed and decrease heat. This is a lot more than just moving a circuit from place A to place B, it will involve different designs too. Then there is the question of how well heat-sinked the sensor is to the rest of the body and how much heat is saved (or generated) overall inside the camera by the new design.... Way! to complex of an issue to discuss in a forum, particularly when we know none of the details....

Yup, but my claim still stands. The more you move stuff to sensor, the more it will generate heat, and enough heat will increase the noise. Of course when they optimize the system, they move only items that improve more noise than the heat degrades.

I think thermal noise is most pertinent to long-exposure photography with extremely low light levels (e.g. Astro)

 

[tpatana]

Re: Patent: Subdividing High & Low Sensitivity Pixels

They (Canon) can get a lot of performance gain by simply moving the signal chain and digitization onto the sensor chip. It would be a lot more conservative (both technically and financially) to do that than it would be to move wholesale to this newly disclosed approach. I believe there is ZERO chance of this making it into a commercial product in the next 12 months. This might be in the 1DX3 or 5D5.

If you move lot of stuff on the sensor chip, it'll heat up and add noise.

If you go to finer lithography you can increase speed and decrease heat. This is a lot more than just moving a circuit from place A to place B, it will involve different designs too. Then there is the question of how well heat-sinked the sensor is to the rest of the body and how much heat is saved (or generated) overall inside the camera by the new design.... Way! to complex of an issue to discuss in a forum, particularly when we know none of the details....

Yup, but my claim still stands. The more you move stuff to sensor, the more it will generate heat, and enough heat will increase the noise. Of course when they optimize the system, they move only items that improve more noise than the heat degrades.

I think thermal noise is most pertinent to long-exposure photography with extremely low light levels (e.g. Astro)

That's bit different.

Basically any electronic signal has noise, and basically any active component will increase noise if you increase the temperature.

So if you move some of the signalling into sensor, and increase the temperature, your noise would also go up. I don't know enough about sensors to say if it'd be 0.00000001% increase, or something significant. So it could be only theoretical increase, or it could be something obvious.

Hmm, has any of the test houses run test with different ambient temperatures? E.g. test exactly same scenario in controlled room with 20C, 30C, 40C and 50C. Compare noise. That could give some idea if the noise would be insignificant or not.

 

[brad-man]

Re: Patent: Subdividing High & Low Sensitivity Pixels

They (Canon) can get a lot of performance gain by simply moving the signal chain and digitization onto the sensor chip. It would be a lot more conservative (both technically and financially) to do that than it would be to move wholesale to this newly disclosed approach. I believe there is ZERO chance of this making it into a commercial product in the next 12 months. This might be in the 1DX3 or 5D5.

If you move lot of stuff on the sensor chip, it'll heat up and add noise.

If you go to finer lithography you can increase speed and decrease heat. This is a lot more than just moving a circuit from place A to place B, it will involve different designs too. Then there is the question of how well heat-sinked the sensor is to the rest of the body and how much heat is saved (or generated) overall inside the camera by the new design.... Way! to complex of an issue to discuss in a forum, particularly when we know none of the details....

Yup, but my claim still stands. The more you move stuff to sensor, the more it will generate heat, and enough heat will increase the noise. Of course when they optimize the system, they move only items that improve more noise than the heat degrades.

and my claim still stands.... insufficient data!

(I think we are saying the same thing from different perspectives)

It will be quite the balancing act to get it right..... You add more circuitry, but make it more efficient... reduce internal heat in one spot, make it worse in another.... I wonder if we will start to see heat pipes from the sensor to the frame? It could be interesting....

To my way of thinking, it is inconceivable that Canon has not been working on something to deal with the DR problems and the noise problems. I would not be surprised in the least to start seeing 16 bit RAW files out of them soon.... something is coming......

You keep using that word...

 

[bdunbar79]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I don't know why you act like putting everything on-die is so scary because it will create more heat and cause thermal noise. Sony has already figured this out and is the reason why their sensor's have lower read noise than Canon sensors. When you put the ADC's off-die you have to get the analog signal there somehow and whatever they're doing (probably pre-amp'ing or mixing) is causing increased read noise. Sony figured out how to put them on-die and minimize thermal noise. This is no longer a technological concern.

 

[jrista]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Guys, if you read the patent, this is not about dynamic range expansion. In the prior art section Canon describes how high and low gain pixels have been used in past patents by other companies for the purposes of expanding dynamic range, however in the description of their own invention, they clearly state the purpose is "shading correction". This patent is about correcting vignetting, not expanding dynamic rage. Based on the text, it sounds like this is a means of dealing with the high incident light angles of large sensors used in mirrorless cameras:

[0003] 2. Description of the Related Art

[0004] Conventionally, there is a known image pickup device that composites outputs of two or more pixels of different areas and outputs the combined signal as a signal of one unit pixel in order to expand a dynamic range (see Japanese Laid-Open Patent Publication (Kokai) No 2010-283573 (JP 2010-283573A)).

[0005] FIG. 10 is a view showing a pixel arrangement of the image pickup device described in the above-mentioned publication.

The part of the patent that relates to dynamic range enhancement is actually a republication of prior art, and is only shown as it was a reference upon which Canon's patent was based. They put the subpixel design from that patent to use to a different purpose, that of vignette correction:

[0007] On the other hand, in an image pickup apparatus, such as a digital still camera, it is known that an output of the image pickup device is affected by shading caused by peripheral darkening of an image pickup optical system, unevenness of sensitivity of an image pickup device, etc. Shading correction may be performed so that brightness of the whole image becomes uniform in order to prevent such shading. For example, a conversion property that converts an image obtained by photographing uniform luminance surface that has uniform luminance distribution into a target output (i.e. an image) is found beforehand. At the time of photographing, shading of an image obtained by photographing is corrected using the conversion property concerned.

The patent proceeds to describe how Canon used different subpixel arrangements for each unit pixel and the aligned subpixel centroids in order to eliminate vignetting at a hardware level, effectively eliminating the need to use software vignette correction. It also proceeds to explain that this hardware process could be simulated by computer software. However it also states fairly clearly that this patent is explicitly related to shading correction only, and outside of the initial mention of a prior patent that uses similar pixel architecture and high/low gain sub pixels to enhance DR, this patent does NOT cover such an approach.

The most interesting part of the patent, to me, is the fact that they describe a highly advanced PER-PIXEL readout architecture, one which includes MULTIPLE ADC units PER-PIXEL:

[0062] A processing circuit for the image pickup device shown in FIG. 7 is provided with an A/D converter for every pixel that constitutes a unit pixel, a high-sensitivity-pixel adding circuit, a shading correction circuit, and a compositing circuit as with the processing circuit shown in FIG. 3. Then, the shading correction circuit performs shading corrections of the high-sensitivity-pixel output, middle-sensitivity-pixel output, and low-sensitivity-pixel output using common shading correction data in the same manner as described with reference to FIG. 4.

I've read about per-pixel ADC before, but never anything on the level of complexity that this patent describes. I have to imagine that Canon is using a much smaller fabrication process to pack this much logic complexity into each pixel. It also means that they would have to move all readout logic onto the sensor die, and with that much parallelism, it's probable they have reduced ADC frequency. That alone should lead to higher DR....however per-pixel ADC involves it's own challenges with noise, so we would have to see actual test results to know whether this actually does improve DR or not.

 

[JMZawodny]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I don't know why you act like putting everything on-die is so scary because it will create more heat and cause thermal noise. Sony has already figured this out and is the reason why their sensor's have lower read noise than Canon sensors. When you put the ADC's off-die you have to get the analog signal there somehow and whatever they're doing (probably pre-amp'ing or mixing) is causing increased read noise. Sony figured out how to put them on-die and minimize thermal noise. This is no longer a technological concern.

+1

Increased dark current at higher temperatures will be seen comparable shot noise only for long exposures. However at long exposures the duty cycle of the ADC section will be low and it should not be generating that much heat. The improved read noise by having this on chip will certainly outweigh any increases in dark current (and associated noise) due to increased power dissipation. This is not that difficult of a trade space to optimize. Besides in that recent interview with (forgot name) from Canon, he said they are doing this exact thing and to expect to see it come to market soon. That was long enough ago for "soon" to be "now".

 

[jrista]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I don't know why you act like putting everything on-die is so scary because it will create more heat and cause thermal noise. Sony has already figured this out and is the reason why their sensor's have lower read noise than Canon sensors. When you put the ADC's off-die you have to get the analog signal there somehow and whatever they're doing (probably pre-amp'ing or mixing) is causing increased read noise. Sony figured out how to put them on-die and minimize thermal noise. This is no longer a technological concern.

+1

Increased dark current at higher temperatures will be seen comparable shot noise only for long exposures. However at long exposures the duty cycle of the ADC section will be low and it should not be generating that much heat. The improved read noise by having this on chip will certainly outweigh any increases in dark current (and associated noise) due to increased power dissipation. This is not that difficult of a trade space to optimize. Besides in that recent interview with (forgot name) from Canon, he said they are doing this exact thing and to expect to see it come to market soon. That was long enough ago for "soon" to be "now".

It is also a simple fact that Sony's sensors, as well as some newer Canon sensors, have RADICALLY lower dark current than older designs. Sony puts most of their circuitry on the sensor die, and they are still able to achieve lower dark current.

The other benefit of moving ADC onto the die is you can increase it's parallelism. These days, column-parallel ADC is pretty standard (except with Canon, sadly), and per-pixel ADC is gaining increased research. According to the patent, it sounds like this subpixel architecture from Canon employs per-pixel ADC, or at least one of the designs called out in the patent does.

With increased parallelism comes lower operating frequency, which also reduces noise. Canon's biggest problem with read noise today is their very high frequency off-die ADC units, which all have to handle millions of pixels per second. Reduce that to having to handle maybe a few hundred pixels per second, and read noise should drop considerably. A reduction in read noise would have the greatest impact on DR for Canon, more than any multi-gain reads might.

 

[Don Haines]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I don't know why you act like putting everything on-die is so scary because it will create more heat and cause thermal noise. Sony has already figured this out and is the reason why their sensor's have lower read noise than Canon sensors. When you put the ADC's off-die you have to get the analog signal there somehow and whatever they're doing (probably pre-amp'ing or mixing) is causing increased read noise. Sony figured out how to put them on-die and minimize thermal noise. This is no longer a technological concern.

+1

Increased dark current at higher temperatures will be seen comparable shot noise only for long exposures. However at long exposures the duty cycle of the ADC section will be low and it should not be generating that much heat. The improved read noise by having this on chip will certainly outweigh any increases in dark current (and associated noise) due to increased power dissipation. This is not that difficult of a trade space to optimize. Besides in that recent interview with (forgot name) from Canon, he said they are doing this exact thing and to expect to see it come to market soon. That was long enough ago for "soon" to be "now".

It is also a simple fact that Sony's sensors, as well as some newer Canon sensors, have RADICALLY lower dark current than older designs. Sony puts most of their circuitry on the sensor die, and they are still able to achieve lower dark current.

The other benefit of moving ADC onto the die is you can increase it's parallelism. These days, column-parallel ADC is pretty standard (except with Canon, sadly), and per-pixel ADC is gaining increased research. According to the patent, it sounds like this subpixel architecture from Canon employs per-pixel ADC, or at least one of the designs called out in the patent does.

With increased parallelism comes lower operating frequency, which also reduces noise. Canon's biggest problem with read noise today is their very high frequency off-die ADC units, which all have to handle millions of pixels per second. Reduce that to having to handle maybe a few hundred pixels per second, and read noise should drop considerably. A reduction in read noise would have the greatest impact on DR for Canon, more than any multi-gain reads might.

+2
long signal paths (relatively) and high frequency help to generate noise on Analog signals... A per-pixel A/D - AGC system on chip might end being the holy grail of clean signal technology.... Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot.... At 2 transistors per bit, that's 1.6 TRILLION transistors on the card.... If you put that many transistors on a sensor you would have 32,000 transistors per pixel on a 50Mpixel sensor..... I bet there is a team at Canon (and Sony) who have already spent several years on the problem....

Adding a bit more logic to the sensor is not going to heat it up appreciably..... adding a lot... who knows? It will be interesting to see where the balance lies....

 

[tpatana]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I don't know why you act like putting everything on-die is so scary because it will create more heat and cause thermal noise. Sony has already figured this out and is the reason why their sensor's have lower read noise than Canon sensors. When you put the ADC's off-die you have to get the analog signal there somehow and whatever they're doing (probably pre-amp'ing or mixing) is causing increased read noise. Sony figured out how to put them on-die and minimize thermal noise. This is no longer a technological concern.

Did you read what I said?

It's game of optimization, certain items on chip decrease noise more than the heat increases, so it'll be better to have it on die. Some other items would hurt more than they help, so those are not moved on the die. Simple as that. So moving ADC helps more than it hurts -> move it.

And "everything on die" is galaxies away from "adc on chip". I can accept calling it everything when there's data-bus coming out from the sensor and you can hook CF/SD card directly on that data bus.

 

[tpatana]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot....

And Samsung SSD SM951 500GB is about size of 2 CF cards, and it get very very hot. People even install aftermarket heat sinks to that. So should either of these examples prove something about camera technology? I don't think so.

Like I said already couple times, I'm sure they'll move the items that make sense moving, but it's still far from moving everything there. Was it 1DC where the extra weight was due to the added heat sinks inside the body? So yes, there's items inside camera that generate plenty heat.

 

[jrista]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot....

And Samsung SSD SM951 500GB is about size of 2 CF cards, and it get very very hot. People even install aftermarket heat sinks to that. So should either of these examples prove something about camera technology? I don't think so.

Like I said already couple times, I'm sure they'll move the items that make sense moving, but it's still far from moving everything there. Was it 1DC where the extra weight was due to the added heat sinks inside the body? So yes, there's items inside camera that generate plenty heat.

The SSD gets hot because it operates at an extremely high frequency. The amount of current flowing through the device matters. The architecture of the circuitry matters. A sensor is a radically, radically different beast than an SSD drive or a CPU. For one thing, CMOS sensors trickle power (one of their benefits over CCDs), and they are basically passive during an exposure. Modern sensor architectures utilize explicit logic to limit dark current flow (not just measuring it, but limiting it), which reduces it even further. Your concerns about heat with increasing on-die transistor count on a sensor are completely unfounded, especially if you are assuming that the kind of heat that accumulates in ultra high frequency use cases like SSD or CPU's applies to sensor technology.

BTW, the primary heat dissipators in digital cameras are not the sensors, not by a long shot. It's the digital signal processors, the DIGICs and EXPEEDs and BIONZ chips that generate significantly more heat than the sensor. The primary reason sensors generate heat is when they are read out. The lower the frequency of sensor reads, the lower the temperature of the sensor. The most taxing thing for a sensor is a video read, however even with extensive video recording, modern high density sensors still don't get much above ambient temperature. The DSP can get a good 10-15C above ambient, though.

 

[tpatana]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot....

And Samsung SSD SM951 500GB is about size of 2 CF cards, and it get very very hot. People even install aftermarket heat sinks to that. So should either of these examples prove something about camera technology? I don't think so.

Like I said already couple times, I'm sure they'll move the items that make sense moving, but it's still far from moving everything there. Was it 1DC where the extra weight was due to the added heat sinks inside the body? So yes, there's items inside camera that generate plenty heat.

The SSD gets hot because it operates at an extremely high frequency. The amount of current flowing through the device matters. The architecture of the circuitry matters. A sensor is a radically, radically different beast than an SSD drive or a CPU. For one thing, CMOS sensors trickle power (one of their benefits over CCDs), and they are basically passive during an exposure. Modern sensor architectures utilize explicit logic to limit dark current flow (not just measuring it, but limiting it), which reduces it even further. Your concerns about heat with increasing on-die transistor count on a sensor are completely unfounded, especially if you are assuming that the kind of heat that accumulates in ultra high frequency use cases like SSD or CPU's applies to sensor technology.

BTW, the primary heat dissipators in digital cameras are not the sensors, not by a long shot. It's the digital signal processors, the DIGICs and EXPEEDs and BIONZ chips that generate significantly more heat than the sensor. The primary reason sensors generate heat is when they are read out. The lower the frequency of sensor reads, the lower the temperature of the sensor. The most taxing thing for a sensor is a video read, however even with extensive video recording, modern high density sensors still don't get much above ambient temperature. The DSP can get a good 10-15C above ambient, though.

Pretty much what I've been trying to say, but some reason people read my text incorrect, and reply based on what they thought I said, not what I actually tried to say.

 

[jrista]

Re: Patent: Subdividing High & Low Sensitivity Pixels

I think I missed the word 'don't' in your post. Now that I see it, yes, were saying the same thing.

BTW, the trend IS to move everything onto the sensor die. That actually includes image processing logic as well, although at the moment sensors and image processors are simply being stacked with minimal trace lengths to connect the two (addressing what don said about noise in long traces.)

Most designs are working towards ever lower power as well, including low heat dissipation, in the image processing logic. Smaller, thinner, lower power, lower heat packages that will ultimately fit all on a single die of size ranging from about 1/8th of an inch in size to full frame in size.

So I would totally expect everything to be on the sensor at some point. I also wouldn't expect that to hurt image quality at all, if it did, people wouldn't do it.

 

[Don Haines]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot....

And Samsung SSD SM951 500GB is about size of 2 CF cards, and it get very very hot. People even install aftermarket heat sinks to that. So should either of these examples prove something about camera technology? I don't think so.

Like I said already couple times, I'm sure they'll move the items that make sense moving, but it's still far from moving everything there. Was it 1DC where the extra weight was due to the added heat sinks inside the body? So yes, there's items inside camera that generate plenty heat.

The speed that it runs at makes a huge difference too.... Adding more circuitry, but running slower... I wonder what the net result would be.

I think we are in agreement... You say "I'm sure they'll move the items that make sense moving" and I say "Adding a bit more logic to the sensor is not going to heat it up appreciably..... adding a lot... who knows? It will be interesting to see where the balance lies...." sort of the same, but I agree that they will move what makes sense to move..

 

[Don Haines]

Re: Patent: Subdividing High & Low Sensitivity Pixels

Consider that you can fit 200GB of memory onto a micro SD card and nobody ever complains of them getting hot....

The SSD gets hot because it operates at an extremely high frequency. The amount of current flowing through the device matters. The architecture of the circuitry matters. A sensor is a radically, radically different beast than an SSD drive or a CPU. For one thing, CMOS sensors trickle power (one of their benefits over CCDs), and they are basically passive during an exposure. Modern sensor architectures utilize explicit logic to limit dark current flow (not just measuring it, but limiting it), which reduces it even further. Your concerns about heat with increasing on-die transistor count on a sensor are completely unfounded, especially if you are assuming that the kind of heat that accumulates in ultra high frequency use cases like SSD or CPU's applies to sensor technology.

This is why I used the MicroSD example.... LOTS!!! of transistors, but at a slow frequency and intermittent use.... kind of like a sensor....

BTW, the primary heat dissipators in digital cameras are not the sensors, not by a long shot. It's the digital signal processors, the DIGICs and EXPEEDs and BIONZ chips that generate significantly more heat than the sensor. The primary reason sensors generate heat is when they are read out. The lower the frequency of sensor reads, the lower the temperature of the sensor. The most taxing thing for a sensor is a video read, however even with extensive video recording, modern high density sensors still don't get much above ambient temperature. The DSP can get a good 10-15C above ambient, though.

I wouldn't be surprised if the DSP got even hotter than that.

I have never had a DSLR overheat, but boy oh boy what a problem with a GoPro! I have had to set up sunshades and mount the little suckers on a heatsink.... Personal opinion here, but I think plastic bodies are terrible, particularly when shooting video.... I like the idea of a metal frame where you can use the tripod mount to connect the camera to an even bigger heatsink.

BTW, Jon, ages ago you gave great advice about the use of hunting blinds and bird photography, as I type this, I am in my hunting blind watching hundreds of ducks and geese, with a steady flow of chickadees to the feeders beside me.....

 

[TAF]

Re: Patent: Subdividing High & Low Sensitivity Pixels

The biggest problem with moving everything onto the die is yield.

Unlike a memory chip, where you can have 20-40% extra bits and map around the bad ones, a camera sensor doesn't offer that opportunity.

I wonder if Canon (or anyone else) has any patents on ways to increase yield, or if they consider that trade secrets and don't talk about it?