Canon publishes a paper discussing a new 3.4 μm pixel pitch global shutter CMOS image sensor with dual in-pixel charge domain memory

Canon Rumors Guy

Canon EOS 40D
CR Pro
Jul 20, 2010
10,779
3,158
Canada
www.canonrumors.com
The Japanese Journal of Applied Physics has published a Canon Inc. paper on a global shutter entitled  “A 3.4 μm pixel pitch global shutter CMOS image sensor with dual in-pixel charge domain memory” by Masahiro Kobayashi, Hiroshi Sekine, Takafumi Miki, Takashi Muto, Toshiki Tsuboi, Yusuke Onuki, Yasushi Matsuno, Hidekazu Takahashi, Takeshi Ichikawa, and Shunsuke Inoue.
From the paper:
In this paper, we describe a newly developed 3.4 μm pixel pitch global shutter CMOS image sensor (CIS) with dual in-pixel charge domain memories (CDMEMs) has about 5.3 M effective pixels and achieves 19 ke− full well capacity, 30 ke−/lxcenterdots sensitivity, 2.8 e- rms temporal noise, and −83 dB parasitic light sensitivity. In particular, we describe the sensor structure for improving the sensitivity and detail of the readout procedure. Furthermore, this image sensor realizes various readout with dual CDMEMs. For example...

Continue reading...
 
  • Like
Reactions: 1 user
Sep 26, 2018
280
420
3.4u pixel pitch would roughly translate to a 32Mp APS-C sensor. However, this device appears to be mostly targeted at video capture and it is either a test chip, or a prototype of a sensor aimed at a prosumer 4K camcorder.

Based on 32MP APS-C, that works out to ~80MP full-frame. That's in the same ballpark as the 70MP rumored sensor.

This could completely solve the rolling-shutter problems that the Canon ILCs suffer from, and also improve silent shutter performance.
 
  • Like
Reactions: 1 user
Upvote 0
Feb 6, 2019
198
112
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?
One more component to break. Good for Canon, kinda bad for us, I think ... not too sure.

What they're talking about is for video. Majority of photogs shoots @ about 1-2fps, relative to motion pictures (30-60fps), w/ a really fast shutter speed (e.g. 1/3000s). Consider the two shutter types (rolling and global shutter), the image quality is critical as you increase the frame rate, but this is only mattered if you put the images together and turn it into motion pictures. Otherwise, this topic is useless for many of us photogs. I believe canon uses a hybrid technology right now, combining electronic and mechanical shutter curtain to capture image (e.g. canon 5D IV ... and EOS R!?!?)

Canon is behind on the video camera side while small company like BlackMagic have produced more proper camera for fast motion picture.
 
  • Haha
Reactions: 1 user
Upvote 0
Jun 20, 2013
2,505
147
And look at what the Sony rumor for the a7IIIS is: global shutter. Blackmagic does it in their Ursa cameras, so, it makes sense for a video-oriented mirrorless camera.

Canon already has a global shutter for video. It's in the C700.
This is a video only sensor, it's not going into a stills camera.
 
  • Like
  • Haha
Reactions: 4 users
Upvote 0
Considering the downsides, would it be more viable to develop electronic shutter that is as fast or possibly even faster than the current mechanical ones?

A fast electronic shutter means that you collect light only over a short period - e.g. 1/2000 second. With limited light you need high ISO or wide apertures. And it does not circumvent the problem that you cannot read out the whole (standard) CMOS chips fast enough for the required framerates as far I understand the necessity to read information line by line from top to bottom for the current frame before you start to read the next first line to build up the next frame.

With the new sensor layout you can maybe use 1/60 second @24 fps for exposure and 1/40 second for readout (just an example, 1/60 s + 1/40 s = 5/120 s = 1/24 s, the time for 1 frame at 24 fps). The benefit of the described design is that the charge buffers (CDMEM) store electrical charge proportional to the amount of light which hit the sensor during exposure + maybe a much faster readout/conversion speed. After exposure the information is "freezed" in the sensor (memory cells) and you can read out the "freezed still image" and convert it into storable "computer numbers" in a very short time before you clear the memory cells for the next exposure.

1/60 s is ~32 times longer compared to 1/2000 s which converts to 5 stops - 5 stops lower ISO or 5 stops narrower aperture which is always helpful if you already are at ISO 3200 with f/1.4 at 1/60s in low light :)
 
  • Like
Reactions: 1 user
Upvote 0
One more component to break. Good for Canon, kinda bad for us, I think ... not too sure.

What they're talking about is for video. Majority of photogs shoots @ about 1-2fps, relative to motion pictures (30-60fps), w/ a really fast shutter speed (e.g. 1/3000s). Consider the two shutter types (rolling and global shutter), the image quality is critical as you increase the frame rate, but this is only mattered if you put the images together and turn it into motion pictures. Otherwise, this topic is useless for many of us photogs.

No it's not. As a strobist I consider global shutter to be the next big innovation I'm looking forward to. Flash sync at any shutter speed without workarounds like HSS or HS.
 
  • Like
Reactions: 5 users
Upvote 0
't.
The TLDR is: 1) This article DOES appear to me to discuss technology FULLY applicable to stills as well as motion picture. 2) The point of the technology seems to be achieved, which is HDR images from a single sensor, whereby the highlight exposure will have the same motion blur as the shadow exposure.

From the Canon News explanation of the technical article, that I feel totally misunderstands that article:



You've overlooked Fig 10. While this article is somewhat couched in video-type language (e.g., fps discussion in section 6), nothing I see rules out "1 frame period" as being a single still photo, as well as one frame of a video. And within that Fig 10's "1 frame period," they have not just two exposures, but effectively MANY interleaved exposures. For the very short blue segments, they're accumulating the impacting photons in counter (memory) 1, while in the alternating very long orange segments, they're accumulating the photons in counter 2.

This results in HDR because memory 1 will have about 128x fewer photons than memory 2. This means when memory 2 is full (totally blown out highlight) memory 1 will just be starting to have enough photons counted to get above random statistical variation. (128x is about 2^7 which is 7*6 dB or 7 stops).

More importantly, in normal HDR the fast exposure is taken all at once. Instead, here it is taken in many slices. In section 5 they give a case study of a 1/67.5th second exposure: 14.815ms. They add up photons in the fast accumulator for 23 microseconds, then the slow one for 3.68 milliseconds. Then fast slow fast slow fast slow fast, which finishes the exposure.

The fast exposure, true, hasn't caught the motion EVERYWHERE as the subject slides from (say) left to right, but has caught it 5 different places. As long as those places aren't farther apart than a pixel or whatever the lens resolution is, they'll meld together seamlessly. (And, I imagine the number of slices isn't fixed at 5, but rather the slice frequency is more likely to be fixed. A far longer exposure would simply have far more slices, and therefore still produce a continuously-blurred highlight exposure.)

This spreads the "fast" exposure of 115 microseconds (about 1/8700th) across the same total 14.815ms as the "slow" exposure. That gives the "Short Exp." in figure 12 the same motion blur as the "Long Exp.," unlike the "Short Exp." in figure 11 which was taken all at once and is so short as to freeze motion. Likewise Figure 13 shows that there aren't jerky or double-image effects that would normally occur in HDR; compare these images to those at the beginning of the article.

Actually I didn't. This paper is about the dual memory architecture of a global shutter sensor. a normal global shutter only has one memory cell per pixel.
For stills dual memory for the most part is entirely unnecessary and impractical. As with any motion, you'll have blurr. it simply can't be done on the sensor without some pretty convoluted brains sitting behind the sensor doing motion detection. while the time between shots is relatively fast, you still have the exposure itself. for instance, a 1/20th of a second exposure would still, in this case, require at least a 1/10th + 1/120th of a second dual exposure saved to the memory. For video, it's easy, they just take two fast exposures and combine them, as they have much more finite control over the shutter speed. it's not the case though with stills. this isn't the first time Canon has done interleaved exposures for Global Shutter. for instance, it's already in production on their C700 video camera. however all their patents and all their background work on this stuff is all to do with video, and will stay with video.
What Canon has to do with stills is far more expensive. It needs to go to a stacked artitecture, where the DR is not impacted by having memory cells at each pixel. Also Dual Pixel AF extremely complicates Global shutter even more. This type of arrangement would need FOUR memory cells for each actual pixel. Ugh.
that's not what this paper is about, nor is what this sensor is about.
 
  • Haha
Reactions: 1 user
Upvote 0
Unless you want HDR with motion blur and without any additional logic trying to stretch the highlight exposure to match the blur of the shadows exposure. WHICH IS THE ENTIRE POINT OF THE PAPER.


Sure. And thanks to the interleaved sandwich partial-exposure method outlined here, you'll have the same blur on your highlight exposure that you do on your shadows exposure.


Read the journal article. It explains EXACTLY how to do it on the sensor, WITHOUT doing motion detection or anything else.
Quote from after Fig. 12: "the moving object is free from both jerkiness degradation and double image degradation without complicated signal processing "


You've completely missed the entire point of the global shutter mechanism: they're controlling the shutter speed down to 23 microseconds: 1/43,478th of a second. How much more finite do you think you need? They show an example of a 1/65th second exposure, and the entire walkthrough of its operation is applicable to a still image as well as a video.

it's an entirely different shutter process for stills than it is for video. but regardless, see what you want out of it. this technology will not make it into a stills camera for Canon.
what you are not considering is the TIME for each exposure - which is from 30s to 1/8000th of a second for stills. if you have a 1s exposure, you're probably effectively taking a .5s exposure and then a 2s exposure to make up the HDR image for a total of 2.5s.
while doing this while you are doing in video, your shutter speed can be finitely more controlled, because, your shutter speed, well is more controlled and far more predictable.
Then you have the complexity of DPAF which aggravates this by a factor of 2.
but really see into it what you want, but there's a reason canon already has global shutter implemented with the single cell memory version of this technology and it's not available on any ILC.
 
Last edited:
  • Like
  • Haha
Reactions: 1 users
Upvote 0
Mar 25, 2011
16,848
1,835
A company does not expose any of their best and latest technology to the public. The information revealed will have been thoroughly vetted to make sure that nothing confidential is revealed. So, I would place zero confidence that this implementation will happen, they may be using some of the ideas revealed in their latest projects, but this is just a snapshot of the past and likely intended to throw off the competition.

We had global shutters 20 years ago using CCD technology, and still have it. The goal is to achieve it with large CMOS sensors. Tiny CMOS sensors are available with a GS, but there are lots of issues whith a large sensor. The dual memory is a trick that can overcome the need for a super computer to read out 50 milliom photosites instantly.
 
  • Like
  • Haha
Reactions: 1 users
Upvote 0
Sep 26, 2018
280
420
A company does not expose any of their best and latest technology to the public. The information revealed will have been thoroughly vetted to make sure that nothing confidential is revealed. So, I would place zero confidence that this implementation will happen, they may be using some of the ideas revealed in their latest projects, but this is just a snapshot of the past and likely intended to throw off the competition.

They absolutely do expose their cutting edge ideas, when they patent them. Disclosure is part of the price for patent protection. I guarantee anything novel in this paper has already been patented. Companies' trade secrets are usually only things that are non-patentable, because there is no legal protection for a trade secret if it is disclosed in any way. Companies may defer patenting certain things until 1) they have an actual product in development, or 2) they want to publish some aspect of the design. But generally they patent quickly unless they worry the patent length wouldn't be long enough to cover the bulk of the product's market availability.

It's unlikely that Canon is doing anything to purposely confuse their competition. The competitors are more interested in what Canon has already announced or brought to market. Their competitors will reverse-engineer many aspects of Canon's designs 1) to see what Canon is doing, and 2) to see if Canon is infringing their patents. Canon does the same to their competitors. Companies may extrapolate certain performance numbers to have some expectation of certain future products, but I doubt they draw much from patents and journal papers. So much gets patented/published that's never brought to market.

Believe me, I work for a cell-phone manufacturer, and it is crazy what it's possible to learn about other chip-makers' designs just from poking them from the outside. Just one example, by running certain targeted benchmarks and putting the chip under thermal analysis, you can figure out what most of the chip die is dedicated to, and compare how much area they're spending on certain functions to what you are. Then you know what you're leading or trailing on in the design. (Actually, you can guess a lot of the design just by seeing the die photo. Certain things like caches stick out like a sore thumb.)
 
  • Like
Reactions: 1 user
Upvote 0
Canon already has a global shutter for video. It's in the C700.
This is a video only sensor, it's not going into a stills camera.
Canon may want to compete with the Sony A9 whose shutter isn't global, but on the way, and does frame rates close to video. If the Canon sports / wildlife camera application and a true merging of video and stills is what they want (for a future mirrorless one series) this might be the direction.
 
Upvote 0

justaCanonuser

Grab your camera, go out and shoot!
Feb 12, 2014
1,031
927
Frankfurt, Germany
You clearly didn't understand the article. While they're talking about taking an exposure in the context of video, nothing about the method requires it to be a frame of video.
I agree. Such a paper in a physics journal is introducing the principle. The authors use video as main application example because the very fast frame rates are overall a much more demanding task. I am pretty sure that Canon will use such sensors as a platform for different camera models (even more as the already do today), since industrial platform technologies are in general much less expensive than making many special parts for special applications. This is even more probable since any differentiation between stills and video cameras makes less and less sense, at least on the sensor (or electronics) side of life ;). It may make sense in future to stick with different classic form factors of camera bodies for stills or for video, but the core electronics, i.e. the sensor-electronic shutter unit, stuffed into those bodies will be about the same, simply because of the pressure to cut costs.
 
  • Like
Reactions: 1 user
Upvote 0