How "raw" are raw files?

[V8Beast]

With all the comparisons of high ISO raw files that's been going on lately, my pea brain needs some clarification on a few issues. My understanding has always been that raw files represent the information that the camera actually captures in its purest form, with no manipulation whatsoever. However, others have mentioned that Nikon bodies "cook" their files by applying in-camera noise reduction. Obviously, at that point a raw file doesn't represent what the camera captured in it's purest form, and therefore does not accurately represent the performance of the sensor. Does Canon do the same thing with its raw files?

This raises a few more issues. If a camera applies noise reduction to its raw files, what's to say that it can't extend dynamic range with in-camera processing as well? I defer whether or not extending DR of raw files via in-camera software is even possible to the real tech heads on here, but if a camera did do this, how would you even know unless a manufacturer disclosed this info in its literature?

On one hand, I can see why you'd want a true, unprocessed raw file as a means of measuring sensor performance. On the other hand, if in-camera processing is sophisticated enough to improved overall IQ to the point where that a photographer can't even tell that in-camera processing was applied in the first place, who really cares?

 

[Spooky]

I'm fairly sure that all manufacturers apply some processing to their raw files, and it varies between models, hence new raw converters for Lightroom and PS to read and load the raw files from new cameras. The raw file is the purest from the point that it is not the compressed, colour spaced, sharpness, contrast, white balance tweaked jpeg file, but it almost certainly has been processed in camera. I wouldn't worry about this as this is what we get and is the best that Canon engineers have deemed we will have and we can't change that unless there is a firmware update.

 

[te4o]

Bering this in mind could we have a list of the most used RAW converters in particular with their strengths and weaknesses?
We had a lot of discussions about generalized ' Raw vs the rest of the world' without much emphasis on the RAW (on everything else though). I'd like to know do you use several RAW converters in your workflow for different shots and for different purposes or just stick to one converter-organizer like LR or Aperture or whatever?

 

[V8Beast]

I wouldn't worry about this as this is what we get and is the best that Canon engineers have deemed we will have and we can't change that unless there is a firmware update.

I'm not worried about it at all. I trust the egghead engineers at Canon have done their homework and created algorithms that produce the best raw files as possible. If each manufacturer applies some processing to all its raw files, then I find it somewhat silly to so heavily scrutinize raw files as if they're a genuine representation of sensor technology.

 

[3kramd5]

There has to be a certain amount of processing. The sensor develops charge, while the raw contains brightness data. The a-d circuitry converts (processes) accumulated charge in each photosite to a relative brightness, and very likely other operations occur at that time (ie BEFORE the data is rastered).

 

[Radiating]

With all the comparisons of high ISO raw files that's been going on lately, my pea brain needs some clarification on a few issues. My understanding has always been that raw files represent the information that the camera actually captures in its purest form, with no manipulation whatsoever. However, others have mentioned that Nikon bodies "cook" their files by applying in-camera noise reduction. Obviously, at that point a raw file doesn't represent what the camera captured in it's purest form, and therefore does not accurately represent the performance of the sensor. Does Canon do the same thing with its raw files?

This raises a few more issues. If a camera applies noise reduction to its raw files, what's to say that it can't extend dynamic range with in-camera processing as well? I defer whether or not extending DR of raw files via in-camera software is even possible to the real tech heads on here, but if a camera did do this, how would you even know unless a manufacturer disclosed this info in its literature?

On one hand, I can see why you'd want a true, unprocessed raw file as a means of measuring sensor performance. On the other hand, if in-camera processing is sophisticated enough to improved overall IQ to the point where that a photographer can't even tell that in-camera processing was applied in the first place, who really cares?

Both Canon and Nikon ALREADY use processing to increase the dynamic range of RAW files. Nikon's processing is several times more advanced than Canon's though.

Nikon cooks the raw files for noise reduction ONLY during long exposures.

Beyond that there is nothing done to the raw files as far as I've been able to find, and real world tests seem to confirm that no noise reduction is applied to the raw files.

 

[V8Beast]

Both Canon and Nikon ALREADY use processing to increase the dynamic range of RAW files. Nikon's processing is several times more advanced than Canon's though.

Very interesting. So is the Sony/Nikon Exmor sensor that much better than Canon's full-frame sensors, or is some of its advantage in DR attributable to better in-camera processing? Is contrast and color reproduction/saturation also tweaked by in camera before the information is recorded as a raw file?

IMHO, all that really matters is the end product, regardless how large of a role in-camera processing and the sensor itself plays in the equation. I suppose the notion that raw images accurately reflect sensor performance is bogus, but not quite as bogus as a jpeg.

 

[grahamsz]

This is a pretty interesting discussion. Given some of the discussions here from simple benchmarking tests, you'd believe that Nikon are absolutely destroying Canon at low ISOs. While I'm certainly impressed with what Nikon can do at 36MP, there are comparison areas where I think the equivalent Canon image is stronger.

I'm sure it'd be possible to design a sensor and preprocessing logic to create really strong results on synthetic benchmarks. It's the engineering equivalent of teaching to the test, and we've seen in in the past with processor and gpu benchmarks on PCs.

I wonder how well a benchmark can control for something like that.

 

[Meh]

There has to be a certain amount of processing. The sensor develops charge, while the raw contains brightness data. The a-d circuitry converts (processes) accumulated charge in each photosite to a relative brightness, and very likely other operations occur at that time (ie BEFORE the data is rastered).

Not really. To refer to the ADR as "processing" isn't quite right and I'm not sure what you mean by "relative" brightness. What are you suggesting it's relative to? The ADC is a straight-forward digitization of the accumulated charge and it's generally believed to be a linear conversion which if correct would suggest there no "processing" per se.

It's well known that Nikon clips the lowest signals just above the noise floor which gives the appearance of a less noisy RAW file but I've never heard they apply any significant/advanced NR to the RAW data.

 

[Meh]

Both Canon and Nikon ALREADY use processing to increase the dynamic range of RAW files. Nikon's processing is several times more advanced than Canon's though.

You appear to state this as a matter of fact. Can you explain further what processing each is doing increase DR of the RAW file, why you believe Nikon is several times more advanced than Canon, and where you're getting your information from?

 

[bvukich]

Nikon cooks the raw files for noise reduction ONLY during long exposures.

They cook the RAW at speeds slower than 1/4.

I don't consider that a particularly long exposure. Some landscape guys would consider that a moderately fast exposure, and astrophotography guys consider that practically "bullet time".

YMMV

 

[3kramd5]

and I'm not sure what you mean by "relative" brightness. What are you suggesting it's relative to?

Charge

 

[Marsu42]

Nikon cooks the raw files for noise reduction ONLY during long exposures.

When I decided if to get the D7000 or the 60D back then, I read a lot about Canon vs. Nikon. While I don't have the exact link (I think it was somewhere on dpreview), a Nikon guy examined the D7000 raw files and compared them to the D90. The result: Esp. on the D7000 the raw files are heavily modified if iso speed increases, and this results in the raw data files to have significantly different sizes on different iso settings. Afaik, it's easier to get "raw" raw data out of the aps-c Canons than out of the D7000.

 

[Meh]

and I'm not sure what you mean by "relative" brightness. What are you suggesting it's relative to?

Charge

Ok but I wouldn't put it that way exactly. When you say "relative brightness" that would typically mean how bright something is compared to something else... either a ratio or difference in magnitude.

 

[Meh]

Nikon cooks the raw files for noise reduction ONLY during long exposures.

When I decided if to get the D7000 or the 60D back then, I read a lot about Canon vs. Nikon. While I don't have the exact link (I think it was somewhere on dpreview), a Nikon guy examined the D7000 raw files and compared them to the D90. The result: Esp. on the D7000 the raw files are heavily modified if iso speed increases, and this results in the raw data files to have significantly different sizes on different iso settings. Afaik, it's easier to get "raw" raw data out of the aps-c Canons than out of the D7000.

I believe the reason for the different file sizes is that higher ISO files are noisier and therefore the RAW files can not be compressed as much as the less noisier files from lower ISO settings. And yes, RAW files are also compressed but only using lossless compression algorithms (rather than jpeg which is a lossy compression scheme).

 

[Marsu42]

I believe the reason for the different file sizes is that higher ISO files are noisier and therefore the RAW files can not be compressed as much as the less noisier files from lower ISO settings

No, that wasn't it - increased file size for higher noise would be self-evident. Actually, as far as I remember it was exactly the other way around: the D7000 uses strong noise reduction that you cannot turn of, thus high iso files are *smaller* than it would be expected from simply reading the sensor data.

But as I said, I don't find the link anymore, it's been a year since and I didn't follow the discussion since I've got a Canon now.

 

[LetTheRightLensIn]

Both Canon and Nikon ALREADY use processing to increase the dynamic range of RAW files. Nikon's processing is several times more advanced than Canon's though.

Very interesting. So is the Sony/Nikon Exmor sensor that much better than Canon's full-frame sensors, or is some of its advantage in DR attributable to better in-camera processing? Is contrast and color reproduction/saturation also tweaked by in camera before the information is recorded as a raw file?

No, that is not true at all. It has better dynamic range because of the sensor read technology, nothing to do with software.

 

[Radiating]

Both Canon and Nikon ALREADY use processing to increase the dynamic range of RAW files. Nikon's processing is several times more advanced than Canon's though.

Very interesting. So is the Sony/Nikon Exmor sensor that much better than Canon's full-frame sensors, or is some of its advantage in DR attributable to better in-camera processing? Is contrast and color reproduction/saturation also tweaked by in camera before the information is recorded as a raw file?

IMHO, all that really matters is the end product, regardless how large of a role in-camera processing and the sensor itself plays in the equation. I suppose the notion that raw images accurately reflect sensor performance is bogus, but not quite as bogus as a jpeg.

The advantage from Nikon is both in hardware AND software from Nikon. Not only do they process their raw files to maximize dynamic range much better than Canon, but they have better hardware. Color is not tweaked but constrast is (to help dynamic range). So Canon files have more contrast.

In-camera raw "cooking" is a serious issue because cameras are tools sold on their performance and trying to fool people by creating deceptive performance means people get a sub-par product. Generally speaking dynamic range, and long exposure noise are the only things that benefit from in camera processing. Beyond that processing degrades the image. The quality of the sensor is the single largest factor determining the level of noise and sharpness at a given ISO.

There was a recent camera system launched, I beleive it was a 4/3rds system which used in-camera sharpening of raw files, they did this so reviewers would think their lenses were sharper than they were which was a handicap of a small sensor. The lenses acheived litelrally impossible sharpness figures and made the impression you were getting L glass for half as much but the real photos didn't look that great. You were actually getting a worse product for more money because of the deception of raw cooking.

This is a pretty interesting discussion. Given some of the discussions here from simple benchmarking tests, you'd believe that Nikon are absolutely destroying Canon at low ISOs. While I'm certainly impressed with what Nikon can do at 36MP, there are comparison areas where I think the equivalent Canon image is stronger.

I'm sure it'd be possible to design a sensor and preprocessing logic to create really strong results on synthetic benchmarks. It's the engineering equivalent of teaching to the test, and we've seen in in the past with processor and gpu benchmarks on PCs.

I wonder how well a benchmark can control for something like that.

Canon and Nikon already cheat very heavily when it comes to camera performance. Thankfully neither do anything that degrades sensor quality and for the most part they don't cook raw files to boost performance. They do however over-rate the ISO of a given camera to help in back to back benchmarks. The over-rating of ISO is so bad that in some cases it's 80%. Meaning your 6400 ISO photos are actually taken at around 3600 ISO. Thankfully both manufacturers do it so it evens out.