Is it noise?

[SeeFGrant]

Hello friends,

I'm still new to post processing and need some help with an image I shot recently. I took a 40min exposure at 10mm, f5.6, ISO200 on my (throughly weathered) T3i. When I got the image into PS CC this I what I saw. These green and red dots covering the entire image.

Also as you can see from the screen grab (for test purposes) I tried seeing what the noise reduction tools in camera raw would do but they did not seem to get rid of the dots.

My question, is it something on my sensor or is it noise? How would I go about fixing this in PS?

If it is something on my sensor it would not surprise me as my camera has been well used and never serviced.

Thanks in advance,

Corey

 

[neuroanatomist]

Hot pixels. Need dark frame subtraction. Search for recent posts by our extremely knowledgable resident astro guru jrista.

 

[Mt Spokane Photography]

Yes, there is a technique built into your camera to help, but help from Jrista is a good idea.

 

[climber]

Next time, take several photos with a 30 sec exposure and blend them manually. You will get less hot pixels during one 30 sec exposure, than during 40 min.

 

[jrista]

Hello friends,

I'm still new to post processing and need some help with an image I shot recently. I took a 40min exposure at 10mm, f5.6, ISO200 on my (throughly weathered) T3i. When I got the image into PS CC this I what I saw. These green and red dots covering the entire image.

Also as you can see from the screen grab (for test purposes) I tried seeing what the noise reduction tools in camera raw would do but they did not seem to get rid of the dots.

My question, is it something on my sensor or is it noise? How would I go about fixing this in PS?

If it is something on my sensor it would not surprise me as my camera has been well used and never serviced.

Thanks in advance,

Corey

Try reading through this thread:

http://www.canonrumors.com/forum/index.php?topic=26264.0

I basically did the same kind of processing on that image as you should do on yours. I use a tool called PixInsight, advanced and powerful, designed for astrophotography. It's about $250, but it's a perpetual license, and you get all updates. If you can't get that, then you can look into DSS (DeepSkyStacker) which can do master dark generation and subtraction for you. (The thing about dark subtraction is it needs to be done on the original RAW data BEFORE debayering...do it after, and it won't quite work right. So you can't do it in Photoshop.)

 

[SeeFGrant]

Try reading through this thread:

http://www.canonrumors.com/forum/index.php?topic=26264.0

I basically did the same kind of processing on that image as you should do on yours. I use a tool called PixInsight, advanced and powerful, designed for astrophotography. It's about $250, but it's a perpetual license, and you get all updates. If you can't get that, then you can look into DSS (DeepSkyStacker) which can do master dark generation and subtraction for you. (The thing about dark subtraction is it needs to be done on the original RAW data BEFORE debayering...do it after, and it won't quite work right. So you can't do it in Photoshop.)

So first off just so I am understanding what happened. What I am seeing in the image is due to strain on the camera from such a long exposure?

Jrista- As one would expect I cannot really afford that right now, mostly because I am saving for a body upgrade. I will certainly look into DSS.

Also secondly, I can fix this with dark subtraction but it has done before I do anything in PS?

Thank you guys. This was my first time trying long exp. I had a friend next to me using the same lens on a 7D and similar settings that got a far different result. Not sure if it was the difference in bodys? I was under the impression the had the same sensor though.

Corey

 

[jrista]

You can fix the hot pixel issue by subtracting a well-built master dark from each individual light frame you may be stacking, BEFORE the images are debayered (demosaiced). Subtraction of a dark any other way will not produce ideal results, and could make noise or artifacts worse (i.e. you may get rings around hot pixels.)

 

[SeeFGrant]

Jrista- I will try and fix it this weekend. I will post up a few more screen grabs when I do so.

I am also camping this Sat so I am excited to try some more long exp. I will try and do better this time, with a shorter shutter and test out the noise reduction settings in my camera.

any other quick tips are greatly appreciated.


Thanks Jrista and all you guys again you have all been a huge help!

Corey

 

[jrista]

If you do proper dark subtraction, you should be able to use exposures as long as you want. I routinely use exposures many minutes long, sometimes as much as 10 minutes long, with my astrophotography. Dark frames need to match the ISO, exposure time, and temperature (within a few degrees C). So long as you do that, you could take 30 second subs or 30 minute subs, it wouldn't matter. The dark frames, when properly combined into a master dark will subtract the thermal signal.

If longer exposures are necessary to get what you want, then go with longer exposures. Just make sure to use the same ISO and exposure time (and as much as you can, match temperature) for the darks.

 

[rfdesigner]

If you do proper dark subtraction, you should be able to use exposures as long as you want. I routinely use exposures many minutes long, sometimes as much as 10 minutes long, with my astrophotography. Dark frames need to match the ISO, exposure time, and temperature (within a few degrees C). So long as you do that, you could take 30 second subs or 30 minute subs, it wouldn't matter. The dark frames, when properly combined into a master dark will subtract the thermal signal.

If longer exposures are necessary to get what you want, then go with longer exposures. Just make sure to use the same ISO and exposure time (and as much as you can, match temperature) for the darks.

Jrista is entirely correct.. but...


I use IRIS which is a pig to drive but contains a "dark optimisation routine". The problem you have is some pixels exhibit a leakage current and so build up charge over time, far more so than other pixels, so they appear brighter than they should be (a hot pixel). That leakage is related to time, temperature and possibly other settings like ISO. What this means is that you are reccomended to keep these things identical between the dark and the light frame, this is nearly impossible, even on advanced astro cameras. The point of the "dark optimisation routine" is that it works out if the hot pixels in the dark frame are too bright or dim compared to the wanted image frame and makes a correction for you.

IRIS is here: http://www.astrosurf.com/buil/us/iris/iris.htm

and the help file for dark optimisation is here, about half way down the page:

http://www.astrosurf.com/buil/iris/tutorial2/doc9_us.htm

Good luck.

 

[jrista]

Iris is doing dark scaling. I don't know exactly how Iris does it, however I suspect it is basic exposure ratio derived from metadata. This is also how DSS does dark scaling.

PixInsight also does dark scaling, however it is very advanced. It makes use of noise evaluation in each and every light frame, and scales the master dark according to the per-light noise evaluation. When using dark scaling like this, it is actually important to use darks that are at least as long, but prefereably longer, than your lights. This results in darks being scaled down, which reduces the random noise in them, which minimizes the increase in random noise of each light. Despite this, if the temperature of the darks deviate too much from the temperature of the lights, your dark subtraction will still not work ideally. The brightest hot pixels may become a little darker than the average noise, and particularly amp glows will usually invert a little, becoming slightly darker.

When it comes to a proper thermally regulated CCD camera (an advanced astro cam), they are actually very easy to create darks for. Same gain, same exposure time, the temperature is regulated by the camera, so you can always just set the temperature you want, and take darks that are the same temp as your lights. This is actually one of the single biggest reasons to get a CCD in the first place...for the regulated cooling. DSLRs actually have less dark current than CCDs these days...but because their sensor temps are not regulated, you can get worse noise performance out of them. The regulated cooling produces very clean, consistent results frame to frame, night to night, for years on end. This makes it very, very possible to get exactly matched darks with a proper astro CCD camera.

 

[rfdesigner]

Iris is doing dark scaling. I don't know exactly how Iris does it, however I suspect it is basic exposure ratio derived from metadata. This is also how DSS does dark scaling.

PixInsight also does dark scaling, however it is very advanced. It makes use of noise evaluation in each and every light frame, and scales the master dark according to the per-light noise evaluation. When using dark scaling like this, it is actually important to use darks that are at least as long, but prefereably longer, than your lights. This results in darks being scaled down, which reduces the random noise in them, which minimizes the increase in random noise of each light. Despite this, if the temperature of the darks deviate too much from the temperature of the lights, your dark subtraction will still not work ideally. The brightest hot pixels may become a little darker than the average noise, and particularly amp glows will usually invert a little, becoming slightly darker.

When it comes to a proper thermally regulated CCD camera (an advanced astro cam), they are actually very easy to create darks for. Same gain, same exposure time, the temperature is regulated by the camera, so you can always just set the temperature you want, and take darks that are the same temp as your lights. This is actually one of the single biggest reasons to get a CCD in the first place...for the regulated cooling. DSLRs actually have less dark current than CCDs these days...but because their sensor temps are not regulated, you can get worse noise performance out of them. The regulated cooling produces very clean, consistent results frame to frame, night to night, for years on end. This makes it very, very possible to get exactly matched darks with a proper astro CCD camera.

Iris does noise scaling based on a minimum entropy evaluation.. it works very well.

From Christians site:

"Iris solves the latter problem by using two approch.

- In the first one, Iris computes an optimal coefficient applied to the dark map for minimizing the noise in a selected area of the science image.
- In the second one, Iris computes an optimal coefficient applied to the dark map for reducing the entropy of the whole science image."

I'm 99% sure IRIS does noise scaling per image frame.

I did get an eval of pixinsight but I didn't like the overall look, bit too processed for my liking, but I did like many of the tools.. anyway, just FYI I've found that even with my 383L+ (setpoint cooling) I need to do noise scaling. Oh yes found that a number of double length darks noticeably outperform twice the number of equal length darks, a good point raised there as it can be quite a time saver.

 

[jrista]

If your light and dark frames on the 383L+ need scaling, then it sounds like the unit is defective. Any CCD with setpoint cooling should be keeping the temperature within a 1 degree range, and usually it's within a 0.1 degree range. A one degree fluctuation is too small to matter for dark subtraction...if your needing scaling, then temps must be differing by several degrees, in which case, your regulation is failing, resulting in oscillating temperatures. Atik should be able to fix that if the unit is under warranty.

As for Iris, good that it uses noise evaluation.

As for PixInsight, PI does not on it's own produce an intrinsic "look" for images processed with it. How your images look when processed with PI is 100% all up to the user. If you didn't like the look of your images, then you didn't like the look of your own processing. ;P

PI takes time to learn. It is significantly more powerful and capable than Iris, and if you give yourself some time to learn it, you can produce significantly better images with it. However, you have to give yourself some time with the program. It took me about six months to learn PI well enough to start producing images I really liked with it. A lot of that was learning how to use previews, the rest was learning how to generate the right kind of masks to use in concert with various tools. Masking is a critical part of successful PI processing.

One of the most common beginner mistakes with PI is to perform noise reduction with insufficient masking, and to use the NR tools in the wrong way or in the wrong order. TGVDenoise is a powerful NR tool, but it is very easy to totally obliterate high frequency noise, leaving behind lower frequencies of noise...that often results in the dreaded "orange skin" look to your background sky. The solution to that is to heavily mask TGV, which attenuates it's effect. You still get a light orange skin, but you don't obliterate the high frequency noise, which makes another tool, MMT (Multiscale Median Transform) work very well to clean up the orange skinning. Again, though, MMT needs to be heavily attenuated, and run with very high settings, to actually work properly.

PI can produce amazing results. Just take a look at my more recent images here:

http://www.astrobin.com/users/jrista/

My two most recent galaxy images used proper NR techniques, and they look significantly better than some of my earlier attempts, which do not use proper NR techniques. Compare my M101 with my M51...namely the background sky. In M101, which I processed early January, when I didn't quite have a handle on proper masking, the orange skinning effect in the background is apparent. In M51, I had a much better handle on masking, and I was able to properly attenuate NR and get a smooth, clean background sky.

PI can be used to great effect, but you do have to give yourself the proper chance to learn it.

 

[alexthegreek]

really??that's 40 mins at 200 iso?If I do that with my 5 year old 500d hot pixels will look like fireworks!

 

[jrista]

really??that's 40 mins at 200 iso?If I do that with my 5 year old 500d hot pixels will look like fireworks!

You might have had much higher temperatures. The number and intensity of hot pixels is temperature dependent, since they are accumulating additional signal from dark current. Dark current doubles around every five degrees C, so if your sensor temperature was even five degrees warmer than SeeFGrant's, then you could have had about twice the visible hot pixels. If you were more than five degrees warmer, thermal noise could have been quite severe.

I've imaged with my 5D III anywhere from -14F to 94F. Below around 20F or so, dark current becomes fairly negligible to non-existent, above 20F to 45F it is there but not a huge problem, above 45F to 70F is can become problematic, and above 70F it is usually too extreme to deal with. Over 90F, and the thermal signal (dark current) is so strong that it actually diminishes dynamic range by a meaningful amount (since it is an additional signal accumulating in the pixels, it's using up charge capacity that would otherwise be used by photon conversions), and the increase in random noise is very obvious. I don't use my camera for astro if the sensor temperature is going to be above 80F, as I worry that kind of use might diminish the cameras lifespan.

 

[alexthegreek]

really??that's 40 mins at 200 iso?If I do that with my 5 year old 500d hot pixels will look like fireworks!

You might have had much higher temperatures. The number and intensity of hot pixels is temperature dependent, since they are accumulating additional signal from dark current. Dark current doubles around every five degrees C, so if your sensor temperature was even five degrees warmer than SeeFGrant's, then you could have had about twice the visible hot pixels. If you were more than five degrees warmer, thermal noise could have been quite severe.

I've imaged with my 5D III anywhere from -14F to 94F. Below around 20F or so, dark current becomes fairly negligible to non-existent, above 20F to 45F it is there but not a huge problem, above 45F to 70F is can become problematic, and above 70F it is usually too extreme to deal with. Over 90F, and the thermal signal (dark current) is so strong that it actually diminishes dynamic range by a meaningful amount (since it is an additional signal accumulating in the pixels, it's using up charge capacity that would otherwise be used by photon conversions), and the increase in random noise is very obvious. I don't use my camera for astro if the sensor temperature is going to be above 80F, as I worry that kind of use might diminish the cameras lifespan.

well I recently did a test indoors at about probably 20 celsius.Kept the apperture constant starting from iso 100 to 3200 and adjusting the shutter only tried to ettr(no lenr).What I found was that, contrary to common belief (when motion blur is not an issue use low iso and long exposure to fill your sensor with signal), at least in my case I had better results at iso 1600.At low isos that dark current noise was not only in shadows but in midtones too!Very bad!