Canon DSLR News

Canon publishes a paper discussing a new 3.4 μm pixel pitch global shutter CMOS image sensor with dual in-pixel charge domain memory

The Japanese Journal of Applied Physics has published a Canon Inc. paper on a global shutter entitled  “A 3.4 μm pixel pitch global shutter CMOS image sensor with dual in-pixel charge domain memory” by Masahiro Kobayashi, Hiroshi Sekine, Takafumi Miki, Takashi Muto, Toshiki Tsuboi, Yusuke Onuki, Yasushi Matsuno, Hidekazu Takahashi, Takeshi Ichikawa, and Shunsuke Inoue.

From the paper:

In this paper, we describe a newly developed 3.4 μm pixel pitch global shutter CMOS image sensor (CIS) with dual in-pixel charge domain memories (CDMEMs) has about 5.3 M effective pixels and achieves 19 ke− full well capacity, 30 ke−/lxcenterdots sensitivity, 2.8 e- rms temporal noise, and −83 dB parasitic light sensitivity. In particular, we describe the sensor structure for improving the sensitivity and detail of the readout procedure. Furthermore, this image sensor realizes various readout with dual CDMEMs. For example, an alternate multiple-accumulation high dynamic range readout procedure achieves 60 fps operation and over 110 dB dynamic range in one-frame operation and is suitable in particular for moving object capturing. This front-side-illuminated CIS is fabricated in a 130 nm 1P4M with light shield CMOS process.

I don’t speak or truly understand the language of physics, but Richard over at Canon News does and has a broken this paper down to make it a bit more understandable for us scientific mortals.

efmshark

EOS M50
Jan 19, 2018
32
16
3.4u pixel pitch would roughly translate to a 32Mp APS-C sensor. However, this device appears to be mostly targeted at video capture and it is either a test chip, or a prototype of a sensor aimed at a prosumer 4K camcorder.
 
  • Like
Reactions: Trey T

Proscribo

EOS 80D
Jan 21, 2015
192
56
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?
 

flip314

EOS 80D
Sep 26, 2018
172
200
3.4u pixel pitch would roughly translate to a 32Mp APS-C sensor. However, this device appears to be mostly targeted at video capture and it is either a test chip, or a prototype of a sensor aimed at a prosumer 4K camcorder.
Based on 32MP APS-C, that works out to ~80MP full-frame. That's in the same ballpark as the 70MP rumored sensor.

This could completely solve the rolling-shutter problems that the Canon ILCs suffer from, and also improve silent shutter performance.
 
  • Like
Reactions: SwissFrank

preppyak

EOS 6D MK II
Oct 18, 2011
975
45
This could completely solve the rolling-shutter problems that the Canon ILCs suffer from, and also improve silent shutter performance.
And look at what the Sony rumor for the a7IIIS is: global shutter. Blackmagic does it in their Ursa cameras, so, it makes sense for a video-oriented mirrorless camera.
 

Trey T

EOS T7i
Feb 6, 2019
98
42
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?
One more component to break. Good for Canon, kinda bad for us, I think ... not too sure.

What they're talking about is for video. Majority of photogs shoots @ about 1-2fps, relative to motion pictures (30-60fps), w/ a really fast shutter speed (e.g. 1/3000s). Consider the two shutter types (rolling and global shutter), the image quality is critical as you increase the frame rate, but this is only mattered if you put the images together and turn it into motion pictures. Otherwise, this topic is useless for many of us photogs. I believe canon uses a hybrid technology right now, combining electronic and mechanical shutter curtain to capture image (e.g. canon 5D IV ... and EOS R!?!?)

Canon is behind on the video camera side while small company like BlackMagic have produced more proper camera for fast motion picture.
 
  • Haha
Reactions: SwissFrank

mb66energy

EOS 6D MK II
Dec 18, 2011
1,233
158
Germany
www.MichaelBockhorst.de
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?
A fast electronic shutter means that you collect light only over a short period - e.g. 1/2000 second. With limited light you need high ISO or wide apertures. And it does not circumvent the problem that you cannot read out the whole (standard) CMOS chips fast enough for the required framerates as far I understand the necessity to read information line by line from top to bottom for the current frame before you start to read the next first line to build up the next frame.

With the new sensor layout you can maybe use 1/60 second @24 fps for exposure and 1/40 second for readout (just an example, 1/60 s + 1/40 s = 5/120 s = 1/24 s, the time for 1 frame at 24 fps). The benefit of the described design is that the charge buffers (CDMEM) store electrical charge proportional to the amount of light which hit the sensor during exposure + maybe a much faster readout/conversion speed. After exposure the information is "freezed" in the sensor (memory cells) and you can read out the "freezed still image" and convert it into storable "computer numbers" in a very short time before you clear the memory cells for the next exposure.

1/60 s is ~32 times longer compared to 1/2000 s which converts to 5 stops - 5 stops lower ISO or 5 stops narrower aperture which is always helpful if you already are at ISO 3200 with f/1.4 at 1/60s in low light :)
 
  • Like
Reactions: SereneSpeed

tmroper

EOS 80D
Sep 22, 2016
148
52
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?
Isn't a "global shutter" the same thing as an "electronic shutter? Global just being the terms video people use?
 
Mar 7, 2019
5
7
One more component to break. Good for Canon, kinda bad for us, I think ... not too sure.

What they're talking about is for video. Majority of photogs shoots @ about 1-2fps, relative to motion pictures (30-60fps), w/ a really fast shutter speed (e.g. 1/3000s). Consider the two shutter types (rolling and global shutter), the image quality is critical as you increase the frame rate, but this is only mattered if you put the images together and turn it into motion pictures. Otherwise, this topic is useless for many of us photogs.
No it's not. As a strobist I consider global shutter to be the next big innovation I'm looking forward to. Flash sync at any shutter speed without workarounds like HSS or HS.
 

SwissFrank

EOS RP
Dec 9, 2018
289
113
The TLDR is: 1) This article DOES appear to me to discuss technology FULLY applicable to stills as well as motion picture. 2) The point of the technology seems to be achieved, which is HDR images from a single sensor, whereby the highlight exposure will have the same motion blur as the shadow exposure.

From the Canon News explanation of the technical article, that I feel totally misunderstands that article:

Does this technology have any stills image practical purpose? Most likely not, because with stills you have a wide-ranging shutter speed, and even then taking double exposures with a stills sensor is hugely problematic with any motion in the scene.
You've overlooked Fig 10. While this article is somewhat couched in video-type language (e.g., fps discussion in section 6), nothing I see rules out "1 frame period" as being a single still photo, as well as one frame of a video. And within that Fig 10's "1 frame period," they have not just two exposures, but effectively MANY interleaved exposures. For the very short blue segments, they're accumulating the impacting photons in counter (memory) 1, while in the alternating very long orange segments, they're accumulating the photons in counter 2.

This results in HDR because memory 1 will have about 128x fewer photons than memory 2. This means when memory 2 is full (totally blown out highlight) memory 1 will just be starting to have enough photons counted to get above random statistical variation. (128x is about 2^7 which is 7*6 dB or 7 stops).

More importantly, in normal HDR the fast exposure is taken all at once. Instead, here it is taken in many slices. In section 5 they give a case study of a 1/67.5th second exposure: 14.815ms. They add up photons in the fast accumulator for 23 microseconds, then the slow one for 3.68 milliseconds. Then fast slow fast slow fast slow fast, which finishes the exposure.

The fast exposure, true, hasn't caught the motion EVERYWHERE as the subject slides from (say) left to right, but has caught it 5 different places. As long as those places aren't farther apart than a pixel or whatever the lens resolution is, they'll meld together seamlessly. (And, I imagine the number of slices isn't fixed at 5, but rather the slice frequency is more likely to be fixed. A far longer exposure would simply have far more slices, and therefore still produce a continuously-blurred highlight exposure.)

This spreads the "fast" exposure of 115 microseconds (about 1/8700th) across the same total 14.815ms as the "slow" exposure. That gives the "Short Exp." in figure 12 the same motion blur as the "Long Exp.," unlike the "Short Exp." in figure 11 which was taken all at once and is so short as to freeze motion. Likewise Figure 13 shows that there aren't jerky or double-image effects that would normally occur in HDR; compare these images to those at the beginning of the article.
 

SwissFrank

EOS RP
Dec 9, 2018
289
113
However, this device appears to be mostly targeted at video capture and it is either a test chip, or a prototype of a sensor aimed at a prosumer 4K camcorder.
Nothing in the original journal article rules out application to still cameras, though they do use video terms throughout.

The article makes clear that it's a bench mock-up of functionality that could be put into a single part in the future.
 

SwissFrank

EOS RP
Dec 9, 2018
289
113
would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones
I believe so. Section 5 talks of being able to cut off an exposure after 23 microseconds, which is 1/43,000th of a second. That would avoid the rolling shutter that even SLR shutters have always had at sufficiently high speeds.
 
  • Like
Reactions: justaCanonuser

SwissFrank

EOS RP
Dec 9, 2018
289
113
Isn't a "global shutter" the same thing as an "electronic shutter? Global just being the terms video people use?
Not quite.

Both the current "electronic shutter" and "global shutter" are implemented electronically. But what we know today as an electronic shutter reads the cells or at least the rows out one by one, and as it's reading rows on one side of the image, rows on the other side of the image are continuing to accumulate photons which will reflect the object being in a different location. That causes weird images in some cases of moving images, and this artifact is called "rolling shutter."

"Global shutter" is also electronic, but the key is that it is able to stop counting new photons across the entire sensor at the same instant. (It would then read them out cell by cell or row by row, and take some time doing so, but their values wouldn't be updating while that's occuring.) No rolling shutter.

Global shutter is only one part of this story, though. The key thing is that they're using the circuitry that gives them global shutter to ALSO give you HDR, and further to get HDR that doesn't have a positional mismatch between the long exposure for shadows and the short exposure for highlights.
 
Last edited:

canonnews

EOS RP
Dec 27, 2017
255
155
Canada
www.canonnews.com
't.
The TLDR is: 1) This article DOES appear to me to discuss technology FULLY applicable to stills as well as motion picture. 2) The point of the technology seems to be achieved, which is HDR images from a single sensor, whereby the highlight exposure will have the same motion blur as the shadow exposure.

From the Canon News explanation of the technical article, that I feel totally misunderstands that article:



You've overlooked Fig 10. While this article is somewhat couched in video-type language (e.g., fps discussion in section 6), nothing I see rules out "1 frame period" as being a single still photo, as well as one frame of a video. And within that Fig 10's "1 frame period," they have not just two exposures, but effectively MANY interleaved exposures. For the very short blue segments, they're accumulating the impacting photons in counter (memory) 1, while in the alternating very long orange segments, they're accumulating the photons in counter 2.

This results in HDR because memory 1 will have about 128x fewer photons than memory 2. This means when memory 2 is full (totally blown out highlight) memory 1 will just be starting to have enough photons counted to get above random statistical variation. (128x is about 2^7 which is 7*6 dB or 7 stops).

More importantly, in normal HDR the fast exposure is taken all at once. Instead, here it is taken in many slices. In section 5 they give a case study of a 1/67.5th second exposure: 14.815ms. They add up photons in the fast accumulator for 23 microseconds, then the slow one for 3.68 milliseconds. Then fast slow fast slow fast slow fast, which finishes the exposure.

The fast exposure, true, hasn't caught the motion EVERYWHERE as the subject slides from (say) left to right, but has caught it 5 different places. As long as those places aren't farther apart than a pixel or whatever the lens resolution is, they'll meld together seamlessly. (And, I imagine the number of slices isn't fixed at 5, but rather the slice frequency is more likely to be fixed. A far longer exposure would simply have far more slices, and therefore still produce a continuously-blurred highlight exposure.)

This spreads the "fast" exposure of 115 microseconds (about 1/8700th) across the same total 14.815ms as the "slow" exposure. That gives the "Short Exp." in figure 12 the same motion blur as the "Long Exp.," unlike the "Short Exp." in figure 11 which was taken all at once and is so short as to freeze motion. Likewise Figure 13 shows that there aren't jerky or double-image effects that would normally occur in HDR; compare these images to those at the beginning of the article.
Actually I didn't. This paper is about the dual memory architecture of a global shutter sensor. a normal global shutter only has one memory cell per pixel.
For stills dual memory for the most part is entirely unnecessary and impractical. As with any motion, you'll have blurr. it simply can't be done on the sensor without some pretty convoluted brains sitting behind the sensor doing motion detection. while the time between shots is relatively fast, you still have the exposure itself. for instance, a 1/20th of a second exposure would still, in this case, require at least a 1/10th + 1/120th of a second dual exposure saved to the memory. For video, it's easy, they just take two fast exposures and combine them, as they have much more finite control over the shutter speed. it's not the case though with stills. this isn't the first time Canon has done interleaved exposures for Global Shutter. for instance, it's already in production on their C700 video camera. however all their patents and all their background work on this stuff is all to do with video, and will stay with video.
What Canon has to do with stills is far more expensive. It needs to go to a stacked artitecture, where the DR is not impacted by having memory cells at each pixel. Also Dual Pixel AF extremely complicates Global shutter even more. This type of arrangement would need FOUR memory cells for each actual pixel. Ugh.
that's not what this paper is about, nor is what this sensor is about.
 
  • Haha
Reactions: SwissFrank

SwissFrank

EOS RP
Dec 9, 2018
289
113
Based on 32MP APS-C, that works out to ~80MP full-frame. That's in the same ballpark as the 70MP rumored sensor.
Don't read too much into this. They're describing a mocked-up mechanism that's spread all over their test bench like electronic spaghetti, not a prototype component that would be a candidate to put in an actual product.

The important part here isn't the size of the pixels they happened to use, nor the exact times in the example, and so on. Instead, it's the general technique of using a global shutter with two memories per pixel to get good HDR even with motion blur, and also some details (the light guides) of how they can get this additional circuitry onto the sensor while not losing photon capture ability.
 
Last edited:

SwissFrank

EOS RP
Dec 9, 2018
289
113
For stills dual memory for the most part is entirely unnecessary and impractical.
Unless you want HDR with motion blur and without any additional logic trying to stretch the highlight exposure to match the blur of the shadows exposure. WHICH IS THE ENTIRE POINT OF THE PAPER.

As with any motion, you'll have blurr.
Sure. And thanks to the interleaved sandwich partial-exposure method outlined here, you'll have the same blur on your highlight exposure that you do on your shadows exposure.

it simply can't be done on the sensor without some pretty convoluted brains sitting behind the sensor doing motion detection.
Read the journal article. It explains EXACTLY how to do it on the sensor, WITHOUT doing motion detection or anything else.
Quote from after Fig. 12: "the moving object is free from both jerkiness degradation and double image degradation without complicated signal processing "

For video, it's easy, they just take two fast exposures and combine them, as they have much more finite control over the shutter speed.
You've completely missed the entire point of the global shutter mechanism: they're controlling the shutter speed down to 23 microseconds: 1/43,478th of a second. How much more control do you think you need? They show an example of a 1/65th second exposure, and the entire walkthrough of its operation is applicable to a still image as well as a video.

Also I'm puzzled why you say "finite control." What are you trying to say?
 
Last edited:

Trey T

EOS T7i
Feb 6, 2019
98
42
No it's not. As a strobist I consider global shutter to be the next big innovation I'm looking forward to. Flash sync at any shutter speed without workarounds like HSS or HS.
Well, that I didn't think of. That would be good for capturing the flash.