July 25, 2014, 02:26:22 PM

Author Topic: Could "dual pixel" help DR (HDR)?  (Read 2597 times)

ScottyP

  • 6D
  • *****
  • Posts: 516
    • View Profile
Could "dual pixel" help DR (HDR)?
« on: August 04, 2013, 10:51:56 AM »
Could Canon (or Magic Lantern) use dual pixel sensors to shoot different exposures to make single-shot HDR to give extended dynamic range? 

The different exposure would have to be done with ISO since shutter and aperture would obviously be set only one way in each shot.

I may not be understanding something fundamental here.
« Last Edit: August 04, 2013, 10:56:10 AM by ScottyP »
Canon 6D; Canon Lenses: EF 70-200 f/2.8L IS II; EF 24-105 f/4 L; EF 85 f/1.8; EF-S 17-55 f/2.8; Canon 1.4x Mk. III T.C.; Sigma Lens: 35mm f/1.4 "Art"

canon rumors FORUM

Could "dual pixel" help DR (HDR)?
« on: August 04, 2013, 10:51:56 AM »

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8249
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #1 on: August 04, 2013, 11:15:51 AM »
I don't think anyone has one to test.  Its possible that there is access to control iso on each half of a pixel but without a camera and many hours of experimentation, I don't know how any one would know.
It doesn't seem likely that ISO of each half of a pixel can be controlled independently unless Canon has something in mind for higher end cameras.

tcmatthews

  • Rebel T5i
  • ****
  • Posts: 115
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #2 on: August 04, 2013, 12:39:07 PM »
By itself it is unlikely.  However it is very likely that Dual pixel necessitated a change in the readout scheme.  Changes in the readout scheme could effect DR.  So for the time being it is best to wait until the tests start coming out.  After it is in production and in the hands of the review sites.

   

6D, 60D, Rebel XS, EF-S 15-85,  EF 70-200f4L, EF 100f2.8L,  Tamron 70-300VC,  Tamaron 28-75f2.8,   EF 40f2.8,  EF 50f1.8, EF 85f1.8, EF 24mmf2.8IS  EOS M 22f2, 18-55, EOS M-EF adapter, Nex6,  SEL50f18,  SEL1018,  DN 30f2.8, adpaters and Fd and M42 lens.

benlanghorne

  • Power Shot G16
  • **
  • Posts: 12
  • Why not visit my website? ;)
    • View Profile
    • Ben Langhorne Photography
Re: Could "dual pixel" help DR (HDR)?
« Reply #3 on: August 04, 2013, 12:57:30 PM »
It's an interesting concept. It would be nice if a simple line of code would enable use of each half taking a different exposure, but I image that some hardware adjustment would also be required
Canon 5D III, 24-70 II, 50 f/1.4, 70-200 f/2.8 IS, 100-300, 100 f/2.8L

http://www.benlanghornephotography.co.uk

rs

  • 6D
  • *****
  • Posts: 580
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #4 on: August 04, 2013, 01:37:12 PM »
I'd imagine there would be some interesting effects in the bokeh, should this be done - presuming a hack along the lines of the 7D/5D3 with alternating lines, you'd get the left half of every pixel exposing for shadows and the right half of every pixel exposing for highlights. As each half sees a different phase, out of focus areas could take on a strange look - for example, an OOF bright light source could end up with one half of the bokeh ball being close to full white, and the other being mid grey (the shadow channel would be blown out).

If they could alternate between the left and right halves for the two exposures, it could alleviate this potential problem.
5D II | 24-70 II | 70-200 II | 100L | 40 | Sigma 50/1.4 | 40D | 10-22 | 17-55 | 580 EX II | 1.4x TC II

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8249
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #5 on: August 04, 2013, 01:40:17 PM »
By itself it is unlikely.  However it is very likely that Dual pixel necessitated a change in the readout scheme.  Changes in the readout scheme could effect DR.  So for the time being it is best to wait until the tests start coming out.  After it is in production and in the hands of the review sites.

   
Big changes in readout, there is a separate processor dedicated to the readout for autofocus, so I'm sure that many experimenters will try to hack it.  I'd think that it is controlled in the firmware, so it may be possible unless Canon took some drastic measures to make it difficult.

BozillaNZ

  • Rebel T5i
  • ****
  • Posts: 129
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #6 on: August 04, 2013, 08:15:59 PM »
That will give you the weirdest bokeh of all time.

Those vertically split pixels takes half image from the lens, each half will has to be different for the phase AF to work. If they are also exposed differently, then the final image will have something like dark left half circle + bright right half circle in the background blur, which would be totally bizarre.
Canon 1D (classic), 1Ds3, 16-35 II, 24-70 II, 70-200 IS II, 1.4x II, 24 II, S50
Gitzo GT2531, Markins TB21, Q10, RRS clamp system, Lee 100x150 filter system
www.flickr.com/photos/bozillanz/

canon rumors FORUM

Re: Could "dual pixel" help DR (HDR)?
« Reply #6 on: August 04, 2013, 08:15:59 PM »

dgatwood

  • 6D
  • *****
  • Posts: 556
  • 300D, 400D, 6D
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #7 on: August 04, 2013, 09:38:52 PM »
That will give you the weirdest bokeh of all time.

Those vertically split pixels takes half image from the lens, each half will has to be different for the phase AF to work. If they are also exposed differently, then the final image will have something like dark left half circle + bright right half circle in the background blur, which would be totally bizarre.

At best I'd expect just a tiny bit of dark fringing at the very edge of the bokeh where it only hits one fractional subpixel and not the other, resulting in approximately half the expected brightness.  I suspect that such artifacts could easily be avoided with some sophisticated digital filtering, though—perhaps something like this:

  • Take one half-image and interpolate the image to calculate approximately what it would look like when shifted by half a subpixel.
  • Convert both images to floating point subpixel values (to add room for additional precision).
  • Do a subpixel-by-subpixel search to replace off-scale high values in the brighter image with scaled-up values from the darker version of the image.
  • Similarly replace any off-scale low subpixel values in the darker image.
  • Do a subpixel-by-subpixel search for in-scale values where the values differ by more than a plausible amount (the sum of the expected maximum noise amplitudes for the chosen ISO settings, presumably) and replace the values in the shifted image with the values from the unshifted image (after scaling the value).  This fixes the crisp edge problem mentioned previously; you have to pick one image to be "truth", so you should pick the one with the least distortion—i.e. the unshifted image.
  • For each subpixel, compute the average of the two images to (at least in theory) lower the noise floor.
  • Convert back to a higher-bit-depth integer value.

And now you have higher dynamic range and lower noise without fringing (albeit at half the original resolution in one direction).

Or if you actually need that extra resolution, you could do the same thing without the averaging step and then repeat the process in reverse without the averaging step, and you should get a plausible approximation of high dynamic range with full resolution, but no noise reduction and a little bit of imprecision in the location of high-energy (crisp) edges that are outside the dynamic range of one set of subpixels or the other.  Not sure how well it would work in practice, but it could be a fun experiment.  :)

Where I'd expect fascinating differences would be in lens flare.

dr croubie

  • 1D X
  • *******
  • Posts: 1397
  • Too many photos, too little time.
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #8 on: August 05, 2013, 12:22:44 AM »
People concerned about "weird bokeh" need not be. Just think about a (non-Xtrans) bayer array, there's only red and blue in alternate lines. Does the bokeh turn out with weird colours? No, it's all interpolated and corrected in software. (Unless you're the type who likes taking pics at 3200ISO of the inside of your lens cap and pushing 5-stops in PP).

Think of using 'dual pixel' the same way that Magic Lantern just hacked the 5/7D to use alternate ISOs on different lines with the dual-readout thingummy. Assuming that you could hack the firmware to allow readout from each individual half-pixel at a time, then I see this working exactly as the ML hack does right now.
That assuming that it's possible in hardware, maybe there's only one readout transistor for each double-half-pixel, in which case this whole discussion is moot.

It might turn into a RAW processing nightmare, but if ML can hack it to produce nice in-camera Jpegs then it might just be popular.

Also, read this. It is definitely possible to get extended DR using funky high/low sampling. The main reason the Fuji wasn't successful was (afaik) the fact that not many (available) programs could decode the RAWs
Too much gear, too little space.
Gear Photos

rs

  • 6D
  • *****
  • Posts: 580
    • View Profile
Re: Could "dual pixel" help DR (HDR)?
« Reply #9 on: August 05, 2013, 05:08:17 AM »
People concerned about "weird bokeh" need not be. Just think about a (non-Xtrans) bayer array, there's only red and blue in alternate lines. Does the bokeh turn out with weird colours? No, it's all interpolated and corrected in software.
The main difference is everything that has gone before (be it monochrome sensors, bayer, x-trans, foveon) has had each and every photosite capturing the same angle of view. This dual pixel AF sensor is different. Each photosite (sub pixel) is only able to see half of the phase.

Imagine for a moment that you could capture an image independently from the left and the right hand phase of this dual pixel sensor (which Canon will obviously not let you do natively). If you've got everything in focus, the two halves will look identical, but if the something is out of focus, it will appear in a different place on the frame in the two images (think of a split image focusing screen).

Instead of a full sized square photodiode behind the microlens, there are two photodiodes - according to the canon marketing material, they are rectangular in shape, together forming a square of the size/shape of a normal photodiode, with a vertical division. This will allow one side to see one phase, and the other side, the other phase. Theoretically if you have a perfect point source of light (bright, about the size of one pixel should it be in focus) out of focus, together the two phases will see a perfect circle of blur. But on its own, the one side would theoretically see just a semi-circle of blur, with a vertical cut off. The other side will see the other half.

This is not going on over the width of just one sub pixel, but could in extreme cases of blur (think 85L wide open at close focusing distances) cover half the frame. Unlike with a bayer sensor and a simple AA filter to blur out neighbouring pixels, there is no way you can re-assemble this data meaningfully if each half is exposed differently, and one half is blown out.

The only way I can picture it working is to do something along the lines of staggering it - one dual pixel having left underexposed, right overexposed, and then the next dual pixel being the opposite way around - left overexposed, right underexposed. That way, some post processing could look at them in pairs of left phase, and check first of all to see if the overexposed channel has blown out (if so, use the neighbours underexposed left phase value). And secondly, do the same for the underexposed channel if it gets down to the noise floor. Without doing that, bokeh would look odd.
5D II | 24-70 II | 70-200 II | 100L | 40 | Sigma 50/1.4 | 40D | 10-22 | 17-55 | 580 EX II | 1.4x TC II

RLPhoto

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 3276
  • Gear doesn't matter, Just a Matter of Convenience.
    • View Profile
    • My Portfolio
Re: Could "dual pixel" help DR (HDR)?
« Reply #10 on: August 05, 2013, 05:41:31 AM »
Didn't the fuji s5 pro have a similar design with a big pixel and a little pixel? It worked then, could it work now?

canon rumors FORUM

Re: Could "dual pixel" help DR (HDR)?
« Reply #10 on: August 05, 2013, 05:41:31 AM »