Question about RAW

[skitron]

My understanding of RAW is that it is simply the data "as-is" read from the sensor and saved on the memory card with no processing applied. However, I found that the color temp as set in a 50d directly affects the color temp of the saved RAW image. Are other in-camera functions applied to RAW as well?

Sorry for such a basic question but I'm new to DSLR and the last time I was serious about cameras was back in the film days. I thought the description of RAW in the manual was pretty straight forward until I tested color temp on it.

[prestonpalmer]

skitron,

A RAW file contains virtually all of the camera and sensor data when it is written to your card.  It is saved with a "loss-less" compression algorithm unlike JPG which actually throws out a TON of image data.

Before I continue, I recommend that you never again shoot a JPG on your camera, set it to RAW and RAW only, no RAW+JPG.

The color temp you set on your camera will change the way the image appears on your computer screen.  When you set a "Color Temp" you are essentially telling the computer that is translating the raw image, just how exactly to display that image, even though ALL the color temp data still exists in the file.  A good habit you can get into now is NEVER changing the white balance setting on your camera, you can change it AFTER you take the photo as long as you are shooting in RAW.  Set your camera to AWB, and leave it there....forever.  When you get your photos onto a computer you can use Lightroom, Aperture, Photoshop, Bridge, or even DPP to adjust the White balance to EXACTLY what you want after you take the photo by selecting a true white or true black.  You will get the most accurate white balance possible with this method, AND you can apply the color temp change to a bunch of photos.  IE, you have 50 photos in a room with florescent lights, you can select all 50 and correct the white balance, if the next bunch of photos are in a incandescent lit reception all, select all those photos and change the color temp to that situation.  This will prevent you from repeatedly "guessing" as to what color temp to set your camera to, and will make ALL your photos more accurate in much less time. I know this concept may be difficult for you to grasp at this time, but trust me when I tell you to leave your White balance alone and shoot in RAW.  Ive taken more than 500,000 wedding pictures and I can tell you what works and what doesn't. Fast and efficient work flow is a HUGE part of my business.

[jeremymerriam]

skitron,

A RAW file contains virtually all of the camera and sensor data when it is written to your card.  It is saved with a "loss-less" compression algorithm unlike JPG which actually throws out a TON of image data.

Before I continue, I recommend that you never again shoot a JPG on your camera, set it to RAW and RAW only, no RAW+JPG.

The color temp you set on your camera will change the way the image appears on your computer screen.  When you set a "Color Temp" you are essentially telling the computer that is translating the raw image, just how exactly to display that image, even though ALL the color temp data still exists in the file.  A good habit you can get into now is NEVER changing the white balance setting on your camera, you can change it AFTER you take the photo as long as you are shooting in RAW.  Set your camera to AWB, and leave it there....forever.  When you get your photos onto a computer you can use Lightroom, Aperture, Photoshop, Bridge, or even DPP to adjust the White balance to EXACTLY what you want after you take the photo by selecting a true white or true black.  You will get the most accurate white balance possible with this method, AND you can apply the color temp change to a bunch of photos.  IE, you have 50 photos in a room with florescent lights, you can select all 50 and correct the white balance, if the next bunch of photos are in a incandescent lit reception all, select all those photos and change the color temp to that situation.  This will prevent you from repeatedly "guessing" as to what color temp to set your camera to, and will make ALL your photos more accurate in much less time. I know this concept may be difficult for you to grasp at this time, but trust me when I tell you to leave your White balance alone and shoot in RAW.  Ive taken more than 500,000 wedding pictures and I can tell you what works and what doesn't. Fast and efficient work flow is a HUGE part of my business.

Before you give advice you really should be positive on your facts.

Just setting the color temp to AWB and forgetting it is not always a wise solution.  RAW doesn't save all the gamuts of color that always completely represent the true colors.  It works sometimes but not all of the times.  This I know from experience shoot all type of jobs.  AWB has its place in a pinch but if you know the true color temp you are dealing with, images ALWAYS have better colors at especially higher ISO if shot properly.  The closer you are to the mark, the better the colors will be when using LR and PS.  AWB does an awful job with tungsten lighting at times and LR doesn't always do a decent job recovering that error.  I shoot a lot of restaurants and I have learned from my mistakes.

[neuroanatomist]

Just setting the color temp to AWB and forgetting it is not always a wise solution.  RAW doesn't save all the gamuts of color that always completely represent the true colors. 

Before you give advice you really should be positive on your facts.

Yes, you should.  Your 'facts' are wrong.  A RAW file does not have a color temperature applied to the data (only the metadata).  RAW image data contains the readouts from individual pixels - each pixel is a monochromatic sensor (it counts photons regardless of their wavelength), with color data coming from the overlying filter in the Bayer mask - but even though that limits the photons counted to those of a particular part of the visible light spectrum, even that 'color' is artificial - pseudocolor, if you will, based on the RAW conversion software knowing which color (R/G/B) mask was over that particular pixel).  Thus, the setting you choose on your camera for the white balance has no effect whatsoever on the image data in the RAW file. 

It's true that at high ISO, dynamic range is compressed and the ability of the camera to accurately capture the full color gamut is compromised - but for a RAW image that results from the lighting conditions and the ISO setting, not the irrelevant WB setting on the camera.

If your white balance setting is affecting your RAW images, it's an indirect effect that could be called operator error (if not error, then a lack of understanding).  Although the WB setting does not affect the RAW image data, it does affect the JPG conversion.  But wait, you say, "I shoot RAW!"  Even if you're shooting RAW, a JPG preview image is embedded into the RAW file, and that JPG image, converted in-camera based on your settings (including WB, picture style, etc.) is what's displayed on your camera's LCD, and also what's used to generate the histogram displayed on your camera's LCD.  So, if you alter your exposure settings based on that preview image/histogram on-camera, that change in exposure will affect your RAW file image data. 

For example, a large red object under mixed but mostly tungsten light, with AWB set on-camera, may not saturate the red channel of your reduced bit-depth JPG preview/histogram, even if the 'real' red channel is actually saturated (real meaning the RAW data from the pixels under the red filters in the Bayer mask).  So, you may feel comfortable bumping your exposure up a little more (ETTR), and end up significantly clipping the red channel in the RAW image.  But that wasn't the WB setting affecting the RAW file, that was you changing the exposure.  This is a situation where knowing how your equipment works, and it's limitations, can help you avoid problems.

Skitron, you are (mostly) correct.  The RAW file is the mostly 'as is' data from the sensor.  The sensor itself is analog, and the analog data (photons stored as an electirc charge in each pixel well) must be converted to digital information - some minimal processing is also applied as part of that process (differs by manufacturer and model, but they don't disclose that information).  Data for each pixel are stored separately in the RAW file (and later must be demosaiced, i.e. the Bayer color mask of red-green-blue filters over the pixels must be interpolated to merge the color channels).  The settings selected during shooting (white balance, picture style, ALO, etc.) are recorded as metadata in the RAW file, but do not affect the image data (one exception to that is highlight tone priority, which does affect the RAW file).  Depending on the software you use on your computer, some or all of those parameters may be applied as defaults to the RAW file during conversion/viewing - Canon's DPP applies Picture Style, ALO, etc.; 3rd party RAW converters (Adobe ACR, Lightroom, Aperture/iPhoto, etc.) read and apply the WB but little else, and in some cases not even the full WB data (for example, with WB set to AWB or Flash, if you use a Canon Speedlite the color temperature of the flash is reported by the flash to the camera and added to the metadata; if the WB was set to Flash, DPP will use the 'custom' color temperature of the light, whereas 3rd party software will just use the Flash default, e.g. 6000K).  But regardless, you can change the WB setting in the RAW file with no penalty to image quality, since you're just changing metadata.

Hope that helps...

[unfocused]

A good habit you can get into now is NEVER changing the white balance setting on your camera, you can change it AFTER you take the photo as long as you are shooting in RAW.

Just to be clear, though, you might want to adjust the white balance setting if you are trying to balance light from more than one source. May not be absolutely necessary, but it can help you judge the colors better and make adjustments, such as adding a gel to the strobe. RAW will let you change the overall color balance of the scene, but it can't do anything about individual light sources.

[gmrza]

A good habit you can get into now is NEVER changing the white balance setting on your camera, you can change it AFTER you take the photo as long as you are shooting in RAW.  Set your camera to AWB, and leave it there....forever. 

This may not work, from a workflow perspective, for a wedding photographer for instance:
A typical job may result in around 800 images.  The closer those images are to perfect, straight out of the camera, the less work you have afterwards.  - After all, your client does not pay you for the time you spend editing in Lightroom/Aperture/DPP etc.

As for shooting JPGs - I can understand why some wedding photographers shoot JPGs, but my wife and I do not subscribe to their point of view.  (They probably don't have UDMA CF cards, and have not realised that storage is cheap.)
If you have a 1D/1Ds, then it makes absolute sense to write RAW to CF and JPG to SD.  That way you have a level of backup in the event of a card failure.

[neuroanatomist]

The closer those images are to perfect, straight out of the camera, the less work you have afterwards.

I'm not a wedding photographer and not under that sort of time pressure for processing images.  But the problem with 'close to perfect' is that it's not perfect...but if it looks too close, human nature will tempt one to not go that last short distance to perfect.  If the WB noticeably off, you'll probably get it right.  But if it's close, you may leave it alone...

[gmrza]

The closer those images are to perfect, straight out of the camera, the less work you have afterwards.

I'm not a wedding photographer and not under that sort of time pressure for processing images.  But the problem with 'close to perfect' is that it's not perfect...but if it looks too close, human nature will tempt one to not go that last short distance to perfect.  If the WB noticeably off, you'll probably get it right.  But if it's close, you may leave it alone...

You've probably hit the nail on the head: it's horses for courses.  Architectural, landscape and commercial work needs to be perfect (i.e absolutely perfect) - so it pays to spend the time on WB.
For a wedding, after your client has selected the 60 or 70 photos from the 400 odd initial proofs, it is worth working on the 60 to 70 photos that make it into the album, more so for that 20"x30" enlargement that your client is paying you hundreds of dollars for.  That is where I think the wedding photogs who shoot JPG will fall short - when it comes to making those large prints RAW gives you the edge to show details you will not see on the screen or in a small print - probably about 1 stop worth more shadow detail (YMMV).
For wedding formals, it is often even worth it to give the bride a white balance card to hold first - that way that framed print that hangs over the mantlepiece will be perfect because you can get the white balance absolutely right in lightroom afterwards.  (Forget trying to do that with JPG.)

[skitron]

Thanks everyone for your replies. I'm a software developer and also have a background in audio so I 'get it' about the data and meta data being seperate, using a lossless format etc and it all makes perfect sense.

So now that I understand that is what is going on, do all of the in-camera processing functions get applied to the RAW's meta data or just some of them? Just trying to get a practical handle on the capabilities so I can put together a workflow that fits me.

Also, are there any differences in processing quality for any given function (say noise reduction for example) between in camera processing and the Canon software?

Thanks again everyone for chiming in, it is very helpful.

[neuroanatomist]

...do all of the in-camera processing functions get applied to the RAW's meta data or just some of them? Just trying to get a practical handle on the capabilities so I can put together a workflow that fits me.

Also, are there any differences in processing quality for any given function (say noise reduction for example) between in camera processing and the Canon software?

They all get written into the metadata, yes, including things like which AF point was selected, etc.  But, the .CR2 format is proprietary, so even though settings are written to the metadata, they cannot necessarily be read by your processing software.  DPP is from Canon, so it can read everything (you can display your AF point for each shot in DPP, to continue with the example).  But things like Picture Style, ALO, etc., are not read/used by 3rd party software (which may offer similar options, e.g. a 'portrait style', but they aren't the same as Canon's, and if you set one in-camera, the 3rd party software won't apply their version by default).  Hope that makes sense.

Yes, there are differences in processing between in-camera and DPP.  In the camera, the RAW file must be converted to JPG (which includes NR, etc.) using only the DIGIC processor which, while sophisticated, is relatively weak (compared to a computer's CPU), and it must be done fast enough to keep up with the frame rate of the camera.  DPP running on your computer has a lot more processing power, and still takes a lot more time - it's doing a better job of the JPG conversion.

[dougkerr]

My understanding of RAW is that it is simply the data "as-is" read from the sensor and saved on the memory card with no processing applied. However, I found that the color temp as set in a 50d directly affects the color temp of the saved RAW image. Are other in-camera functions applied to RAW as well?

The white balance color correction setting in the camera does not have any effect on the raw data.

However, that setting is saved (in a technically-detailed way) in the metadata of the raw file.

Then,  when the raw data is "developed" in external software to a JPEG or TIFF file, color correction is then applied, as set by the operator in any of various ways. One option is for the operator to tell the software to use the color correction that was set in the camera (which the software can read from the metadata) - the same color correction that the camera would have (or did) use in the in-camera generation of a JPEG file for the shot. But that is an explicit choice by the operator.

Best regards,

Doug

[te4o]

May I through in my questions about RAW too:

Are Picture Settings (Landscape, Portrait etc, i.e. in-camera sharpness, contrast etc. adjustments) applied in RAW or is this only for JPEG (i.e. heavier DIGIC-processed files) ?

Importing RAW in any RAW editor applies certain settings (software manufacturers work on this every day, skitron). Now, once imported in one RAW editor (DPP f.i.) can I get the original RAW file (as delivered from the camera) back out of the editor (not from the CF) like an export and import this "precious original Canon CR.2 RAW" into another RAW-editor (like LR, or Aperture). I know, this doesn't make me a better photographer, but would be interesting in changing RAW-editing systems and reworking some files.

Thanks

[prestonpalmer]

neuroanatomist, thanks for the support and additional RAW info!

gmrza, I completely disagree with your theory on white balance and that "getting close to perfect" is better for workflow.  I AM speaking from a workflow and experience perspective here!  I shoot 40 or so weddings per year in addition to engagement shoots and other photo work, and delivering a $4,000 wedding shoot to a client in 10 days or less starting with 1800 or so unprocessed photos is truly a demanding task. Ive taken OVER 500,000 wedding photos, I can assure you, my workflow is EXTREMELY EFFICIENT! My photo editors all agree, leaving the AWB ALONE on default makes for MUCH MUCH quicker and more accurate post processing.  It simply takes too much work chasing the WB and still having to correct it in the end anyway. As neuroanatomist pointed out, the JPG's "false" representation on the back of your camera doesn't help you anyway as you try to CHASE the right color temp.  Remember the back of the camera is lying to you anyway.  While the average photographer is fiddling with white balance in situations that are changing minute to minute, I am off taking beautiful photos, getting all the shots the fiddler is missing.  AND in the end my product will be superior. every. single. time!

[NormanBates]

not long ago I was told that RAW gets you a better RAW than RAW+JPG

I think that's wrong, but maybe someone here knows why this person had that idea in his mind; my guess is that it might have been so, maybe, in something like "nikon cameras from 1999", or some other early step in the evolution of RAW

anybody?

[jeremymerriam]

Just setting the color temp to AWB and forgetting it is not always a wise solution.  RAW doesn't save all the gamuts of color that always completely represent the true colors. 

Before you give advice you really should be positive on your facts.

Yes, you should.  Your 'facts' are wrong.  A RAW file does not have a color temperature applied to the data (only the metadata).  RAW image data contains the readouts from individual pixels - each pixel is a monochromatic sensor (it counts photons regardless of their wavelength), with color data coming from the overlying filter in the Bayer mask - but even though that limits the photons counted to those of a particular part of the visible light spectrum, even that 'color' is artificial - pseudocolor, if you will, based on the RAW conversion software knowing which color (R/G/B) mask was over that particular pixel).  Thus, the setting you choose on your camera for the white balance has no effect whatsoever on the image data in the RAW file. 

It's true that at high ISO, dynamic range is compressed and the ability of the camera to accurately capture the full color gamut is compromised - but for a RAW image that results from the lighting conditions and the ISO setting, not the irrelevant WB setting on the camera.

If your white balance setting is affecting your RAW images, it's an indirect effect that could be called operator error (if not error, then a lack of understanding).  Although the WB setting does not affect the RAW image data, it does affect the JPG conversion.  But wait, you say, "I shoot RAW!"  Even if you're shooting RAW, a JPG preview image is embedded into the RAW file, and that JPG image, converted in-camera based on your settings (including WB, picture style, etc.) is what's displayed on your camera's LCD, and also what's used to generate the histogram displayed on your camera's LCD.  So, if you alter your exposure settings based on that preview image/histogram on-camera, that change in exposure will affect your RAW file image data. 

For example, a large red object under mixed but mostly tungsten light, with AWB set on-camera, may not saturate the red channel of your reduced bit-depth JPG preview/histogram, even if the 'real' red channel is actually saturated (real meaning the RAW data from the pixels under the red filters in the Bayer mask).  So, you may feel comfortable bumping your exposure up a little more (ETTR), and end up significantly clipping the red channel in the RAW image.  But that wasn't the WB setting affecting the RAW file, that was you changing the exposure.  This is a situation where knowing how your equipment works, and it's limitations, can help you avoid problems.

Skitron, you are (mostly) correct.  The RAW file is the mostly 'as is' data from the sensor.  The sensor itself is analog, and the analog data (photons stored as an electirc charge in each pixel well) must be converted to digital information - some minimal processing is also applied as part of that process (differs by manufacturer and model, but they don't disclose that information).  Data for each pixel are stored separately in the RAW file (and later must be demosaiced, i.e. the Bayer color mask of red-green-blue filters over the pixels must be interpolated to merge the color channels).  The settings selected during shooting (white balance, picture style, ALO, etc.) are recorded as metadata in the RAW file, but do not affect the image data (one exception to that is highlight tone priority, which does affect the RAW file).  Depending on the software you use on your computer, some or all of those parameters may be applied as defaults to the RAW file during conversion/viewing - Canon's DPP applies Picture Style, ALO, etc.; 3rd party RAW converters (Adobe ACR, Lightroom, Aperture/iPhoto, etc.) read and apply the WB but little else, and in some cases not even the full WB data (for example, with WB set to AWB or Flash, if you use a Canon Speedlite the color temperature of the flash is reported by the flash to the camera and added to the metadata; if the WB was set to Flash, DPP will use the 'custom' color temperature of the light, whereas 3rd party software will just use the Flash default, e.g. 6000K).  But regardless, you can change the WB setting in the RAW file with no penalty to image quality, since you're just changing metadata.

Hope that helps...

Help me understand this then.  When i shoot with the wrong white balance setting (lets say its set at AWB when the lighting is tungsten), why is there less play (fine tuning) room in lightroom when the image is overly red?  I know what you said is supposidly right but getting the right color balance is much more difficult and always feeling muddier than when i get the right WB.  In lightroom, it will go from being real warm to real cold in just a small (shiftkey + downarrow) move.

My processing goes much faster and to me looks better when I am closer to the mark.  Maybe the original 5d isn't as accurate as the mark2 AWB.  AWB is always way off on my camera in tungsten situations.