Sony Announces Addition of Uncompressed 14-Bit RAW Still Image Capture for New A Cameras

[jrista]

The thing that annoys me here is the use of the word "flawed". The solution is not flawed. A flawed solution would be a non-working solution. A non-functional solution. A solution that could not actually work. This is the kind of bullS___ you spout, and the reason I cannot stand you. You carefully choose words to twist facts.

Sony SOLVED a problem.

I'm not even sure that's true. Lossy compression wasn't a problem; it functioned exactly as Sony intended it to. They merely added a new option.

I think they took the shortest path - not compressing at all - in order to get that option out there quickly. Hopefully with more time they can develop a lossless compression algorithm.

Aye. I agree.

If there is a limitation in the current hardware that limits them from using a lossless option, then I would expect future models to have better options. It won't take terribly long, the turnaround cycle on the A7 series, the alpha series in general, is pretty short. I used to think that was a problem, but I actually like it, a lot. You don't have to wait three years to get an upgrade if you want/need it.

 

[quod]

Re: So this is big news: Uncompressed 14-bit RAW for new Sony A7 series cameras

I think this shows that Sony is failing to understand and respond to customers' needs...

You mean like how Canon fails to under and respond to its customers' needs for more dynamic range and clean shadow recovery? Are those the kinds of needs we are talking about? Kind of like finding someone lost in the desert who is severely parched and asking for a drink, so you hand them a bottle of whiskey from your pack which will only dehydrate them further, instead of the full water bottle hanging from your belt. Good job, Canon.

Oh, and thank you for the quote, Neuro. You are very kind in providing the words that encapsulates Canon's response to its customers' needs with your own brand of sarcasm and arrogance. :

 

[msm]

The thing that annoys me here is the use of the word "flawed". The solution is not flawed. A flawed solution would be a non-working solution. A non-functional solution. A solution that could not actually work. This is the kind of bullS___ you spout, and the reason I cannot stand you. You carefully choose words to twist facts.

Sony SOLVED a problem.

I'm not even sure that's true. Lossy compression wasn't a problem; it functioned exactly as Sony intended it to. They merely added a new option.

I think they took the shortest path - not compressing at all - in order to get that option out there quickly. Hopefully with more time they can develop a lossless compression algorithm.

If you had no problem with the lossy compression stick with it. Whether it is losslessy compressed or uncompressed will make no difference to you, both options will only waste your hard drive space.

This fix is for those who push their files hard who would in some rare occasions see posterisation along extremely high contrast borders along the vertical axis when they push their files hard. Now they can switch to a lossless format in those situations.

 

[3kramd5]

The thing that annoys me here is the use of the word "flawed". The solution is not flawed. A flawed solution would be a non-working solution. A non-functional solution. A solution that could not actually work. This is the kind of bullS___ you spout, and the reason I cannot stand you. You carefully choose words to twist facts.

Sony SOLVED a problem.

I'm not even sure that's true. Lossy compression wasn't a problem; it functioned exactly as Sony intended it to. They merely added a new option.

I think they took the shortest path - not compressing at all - in order to get that option out there quickly. Hopefully with more time they can develop a lossless compression algorithm.

If you had no problem with the lossy compression stick with it. Whether it is losslessy compressed or uncompressed will make no difference to you, both options will only waste your hard drive space.

Harddrive space is cheap cheap cheap. I will likely continue shooting lossy raw for most everything, but if I happen to be somewhere with conditions which may contribute to artifacting, I'll happily double my file size.

Or I might just shoot uncompressed all the time so I don't have a 'whoops' moment.

 

[AvTvM]

Or I might just shoot uncompressed all the time so I don't have a 'whoops' moment.

Feelin' whoopy?

Fact is, it took Sony two full years to address the problem and the solution they offer is flawed .. or half-assed. Either way, not good enough.

And Canon is even worse in not truly listening to its customers. Full-assed, so to speak.

 

[3kramd5]

Or I might just shoot uncompressed all the time so I don't have a 'whoops' moment.

Feelin' whoopy?

Thinking about it this way:

Whatever camera I use, I'm always shooting raw. For family gatherings and chasing my kid around the house, could I get away with L jpeg? Probably. M jpeg? Probably. S jpeg? Maybe, if I don't print.

But I don't, primarily so that I don't have a brainfart and go to a critical shoot with the thing set to something other than RAW.

Flash memory is a concern, but it's light and not terribly expensive - I can double the capacity I have for the A7R2 for 100 bucks, and I can take cards away from my 5D3 stash.

Hard drive space is not a concern - I can get a 6TB drive at $37/TB.

Cloud space is not a concern - I pay 60 bucks per year for unlimited storage.

Would I prefer lossless? Yep. And hopefully there isn't a hardware limitation, or major performance hit, and will added in a future update.

 

[neuroanatomist]

The thing that annoys me here is the use of the word "flawed". The solution is not flawed. A flawed solution would be a non-working solution. A non-functional solution. A solution that could not actually work. This is the kind of bullS___ you spout, and the reason I cannot stand you. You carefully choose words to twist facts.

So a diamond with a flaw has no value? A flaw evident in a person's personality doesn't mean I can't respect that person. I respect you, for example.

The thing that is a load of crap is to call the addition of uncompressed RAW a "flaw". It may not be the most ideal solution, which probably would have been lossless compression, but it is certainly not flawed.

Sony's uncompressed RAW functions well for you because a one shot buffer meets your needs, and therefore it's not a flawed solution. But if some other feature of a different product meets the needs of some but does not meet your needs, say......Canon's low ISO DR – well, that means Canon delivers "poor, sub-par, unacceptable IQ." Nope, that's not a load of crap at all. :

For the record:

We suspect...

So you consider DPR's speculation to be part of 'the record'. Nope, that's not a load of crap either.


[quote author=jrista"]
Anyway. Good to see this place hasn't changed.
[/quote]

Thanks for your notable contributions in that area.

 

[3kramd5]

The thing that is a load of crap is to call the addition of uncompressed RAW a "flaw". It may not be the most ideal solution, which probably would have been lossless compression, but it is certainly not flawed.

Sony's uncompressed RAW functions well for you because a one shot buffer meets your needs

Where are you reading that an uncompressed RAW fills the buffer? Seems... beyond unlikely, as the files are only ~twice as large.


Edit: never mind, I thought it through and suspect you're saying that 1-shot is good enough for what jrista shoots, so no matter what the capacity ends up, he's good.

 

[Mt Spokane Photography]

Oh what a load of crap. But, not surprising from you.

Lossless compression MIGHT save you about 10 megs under the best of conditions for a high resolution sensor.

That's a pipe dream. Canon and Nikon Lossless compression vary from about 2:1 to 3:1.

Its easy to check, so no need to make a totally incorrect statement.

Open a NEF or CR2 with lossless compression in Photoshop and it tells you the uncompressed file size.

For example, a ISO 100 CR2 Raw from my 5D MK III has a compressed file size of 21.0 MP (Varies by detail). Uncompressed, it is 60.2MP, for a 3:1 compression rate. Or, open the file in DPP, and save it as a 16 bit tiff (14 bit is not a option). The resulting uncompressed 16 Tiff file is 120.4MB.

OK, now save it as a 8 bit (Not 14 bit uncompressed tiff) a lot smaller, only 60.3 MB!

Where in the world did you come up with a 10MB difference?

Try saving the file as a 8 bit uncompressed tiff from photoshop, now its 60.1 MB, slightly better than DPP.

 

[neuroanatomist]

Where are you reading that an uncompressed RAW fills the buffer? Seems... beyond unlikely, as the files are only ~twice as large.

Edit: never mind, I thought it through and suspect you're saying that 1-shot is good enough for what jrista shoots, so no matter what the capacity ends up, he's good.

The latter - he stated essentially just that:

It's entirely subjective. Whether you need a large buffer entirely depends on what your photographing. For landscapes, it doesn't matter a wit to me. I have countless gigs of memory cards, and I only take a frame every 10-15 seconds tops, probably more like every minute or two.

Apparently, the need for buffer depth is subjective and 'depends on what your you're photographing,' but that doesn't apply to low ISO DR.

 

[Mt Spokane Photography]

DPR now has reported on the Sony A7R II cameras they have tested with no compression. The images they have seen are about double the size, or a 40+ MP hit. Apparently Sony says that the buffer will fill faster, but there has been no performance testing as of yet. Presumably, it will take longer to empty the buffer and upload to a computer for editing. Beyond that point in the process, it should not make much if any difference.

The files definitely look better.

http://www.dpreview.com/articles/6144418951/what-difference-does-it-make-sony-uncompressed-raw

It sounds like testing will be done once actual production firmware is released.

 

[Orangutan]

Unsurprising as your

Is this really necessary?

Perhaps like Sony, you're not addressing the actual problem...

Which is....?

 

[Orangutan]

Unsurprising as your

Is this really necessary?

Look at his posting history and you'll understand that for him, it seemingly is.

A look at his posting history also shows him to be very knowledgeable and generous with his time and expertise. He's quite capable of supporting his positions very well without being abrasive; it's a mystery to me why he doesn't play to his strengths.

 

[jrista]

Oh what a load of crap. But, not surprising from you.

Lossless compression MIGHT save you about 10 megs under the best of conditions for a high resolution sensor.

That's a pipe dream. Canon and Nikon Lossless compression vary from about 2:1 to 3:1.

Its easy to check, so no need to make a totally incorrect statement.

Open a NEF or CR2 with lossless compression in Photoshop and it tells you the uncompressed file size.

For example, a ISO 100 CR2 Raw from my 5D MK III has a compressed file size of 21.0 MP (Varies by detail). Uncompressed, it is 60.2MP, for a 3:1 compression rate. Or, open the file in DPP, and save it as a 16 bit tiff (14 bit is not a option). The resulting uncompressed 16 Tiff file is 120.4MB.

OK, now save it as a 8 bit (Not 14 bit uncompressed tiff) a lot smaller, only 60.3 MB!

Where in the world did you come up with a 10MB difference?

Try saving the file as a 8 bit uncompressed tiff from photoshop, now its 60.1 MB, slightly better than DPP.

This is totally incorrect. Photoshop tells you the uncompressed size for RGB DATA!!!!

RAW images store a SINGLE color value per PIXEL. Pixel being directly correlated to the physical element on the sensor. When Photoshop opens an image, it generates an in-memory RGB-per-pixel bitmap. If you loaded the image as 8-bpp, then that is a total of 24-bits per pixel, rather than 14-bits. When you account for the compression, THAT is where you get the 2:1 to 3:1 "ratio".

Same thing goes for saving an image as TIFF. The TIFF image is again RGB, so three color values per pixel. Throw in the increase to 16-bit, and you are radically inflating the actual necessary data sizes here.

I came up with an ~10mb difference by calculating the actual necessary data storage to contain ONE 14-bit data value per sensor pixel, and compared that to actual file sizes. I just looked them up online. As I said, it's very simple math.

 

[jrista]

The thing that is a load of crap is to call the addition of uncompressed RAW a "flaw". It may not be the most ideal solution, which probably would have been lossless compression, but it is certainly not flawed.

Sony's uncompressed RAW functions well for you because a one shot buffer meets your needs

Where are you reading that an uncompressed RAW fills the buffer? Seems... beyond unlikely, as the files are only ~twice as large.


Edit: never mind, I thought it through and suspect you're saying that 1-shot is good enough for what jrista shoots, so no matter what the capacity ends up, he's good.

I am not quite sure what the issue here is either. The Sony A7r II has a 24-frame buffer. With continuous shooting, it fills up in 5 seconds, and it will keep taking one shot at a time forever (or until your memory card fills up). The 5Ds has a 12-frame buffer. So, even IF the uncompressed raw actually does halve the buffer...it will still be as good as the 5Ds.

Neuro, how exactly is this "flawed"? It's no worse than the 5Ds "at worst", and if you don't care about the compression (for anything other than astro, it doesn't matter to me, I was perfectly happy with the lossy compressed images I got from the A7s), the buffer is twice as big! Despite the huge resolution.

Spokane, while I understand what he is trying to do, is actually very flawed in his approach to testing compression ratio, because he is not accounting for the fact that photoshop's in-memory bitmaps store THREE values per pixel, rather than one. So, even with the lowest precision data, 8-bpp, his comparisons are assuming 24-bits per pixel, which is what Photoshop is basing those memory load readings off of.

The image sizes are not going to be anywhere close to 100mb per image. I calculated 73mb myself, however I did not account for the border of masked pixels. Throwing those in, we might have 78-80mb per uncompressed image. This isn't something egregious here. Yeah, it's double the size of a LOSSY compressed image, because lossy compression is how Sony is achieving the 2:1 ratio in the first place. LossLESS compression isn't going to get anywhere near as close to that ratio...not unless your image is packed full of nearly identical pixels that give the algorithm a ton of repeatable data to work with. The noisier the image, the more difficult it is to compress with lossless compression, hence the reason that most cameras that already use it end up approaching the uncompressed image size as you increase ISO (or otherwise encounter increased noise) anyway. And those uncompressed raw image sizes are not two to three times larger than the compressed images. They may be 20-30% larger AT MOST, if that. There isn't a lot of processing power in a DSLR, not compared to personal computers, tablets and phones anyway. The algorithms have to work within the limitations of the hardware, so we are not talking cutting edge compression algorithms here.

I don't know why this is being blown so out of proportion and so twisted like this. You guys act like Sony flubbed. They delivered on a customer demand. They improved something that was lacking in their cameras. The uncompressed images may not be as small as lossless compressed images, but lossless compression is not going to be anywhere close to the ~40mb file size that their current lossY compressed images are. Nowhere even remotely close.

The only conclusion here is that this is just the same old unrelenting die-hard Canon fanboyism. Canon has been holding out on you guys on plenty of things for years. I just read an article on the front page of this very site where one of the top requests from Canon customers is STILL to have af-point linked metering. A feature that has been relegated to the 1D line for decades, and a feature that Canon customers clearly have a broad need and use for, have for decades, and are STILL WAITING. Oh, but it's egregious that Sony took "two years" to fix the lossy compression? It took over a year for complaints about it to mount to a big enough level that Sony prioritized it. You guys are all smart enough to understand how this works. In the grand scheme of things, compared to how long Canon has dragged their feet on BIG customer demands (oh, dare I mention it...dynamic range?), two years is lightning speed in comparison.

Bah. This community...one hell of a WEIRD community...

 

[meywd]

In the name of the CR community I welcome your return, jrista and neruo, we missed your posts.

Back on the topic, why talk theory when it can be tested, let's wait for the tests to see the actual size difference, as for the frame rate hit, the Sony A series aren't meant for action shots, neither the AF nor the lenses help in that regard.

 

[neuroanatomist]

I am not quite sure what the issue here is either. The Sony A7r II has a 24-frame buffer. With continuous shooting, it fills up in 5 seconds, and it will keep taking one shot at a time forever (or until your memory card fills up). The 5Ds has a 12-frame buffer. So, even IF the uncompressed raw actually does halve the buffer...it will still be as good as the 5Ds.

Neuro, how exactly is this "flawed"? It's no worse than the 5Ds "at worst", and if you don't care about the compression (for anything other than astro, it doesn't matter to me, I was perfectly happy with the lossy compressed images I got from the A7s), the buffer is twice as big! Despite the huge resolution.

It's flawed because it needlessly increases file size and that has a negative consequence on buffer depth. People have been asking for a multiple destination feature for the Google Maps mobile app. What if Google delivered a 'solution' that provided directions which needlessly added several miles and several minutes of travel time to every route when the multiple destinations feature was used? Apparently you'd call that a perfectly acceptable 'solution', because you'd still get to your destinations, you're never in a hurry and you can afford the gas...and after all the iOS native Maps app doesn't even offer multiple destinations. I'd call it a flawed solution.

Bah. This community...one hell of a WEIRD community...

Well, you're right about that. Statements like 'Canon sensors deliver poor IQ' definitely add to that WEIRDNESS.

 

[3kramd5]

The image sizes are not going to be anywhere close to 100mb per image. I calculated 73mb myself, however I did not account for the border of masked pixels. Throwing those in, we might have 78-80mb per uncompressed image.

I figure 42,000,000pix * 14-bit/pix *1byte/8bit *1MB/(2^20)byte. 70.1MB of luminance data. Plus whatever additional information a file contains (exif, JPEG preview, etc). I could easily see the 81MB dpreview referred to.

With lossless compression, I imagine the files would end up ~60MB. When the firmware comes out I'll compress some to find out. I do expect more than a 10MB advantage, however. The D800 saves about 33MB between uncompressed 14-bit and losslessly compressed 14-bit .nef.

It's flawed because it needlessly increases file size and that has a negative consequence on buffer depth.

Engineering = tradeoffs.

Uncompressed depletes available buffer, but perhaps lossless compression would overly (one might says "needlessly") tax the processor and cause heat-related noise issues or impact battery life. Who knows?

From the above link, when selecting uncompressed, the D800 has a 16-frame buffer for 74.4MB files. When selecting lossless compressed, the D800 has a 17-frame buffer for 41.3MB files. So a 44% reduction in file size only gains you an additional 5% worth of buffer headroom. I won't claim to know exactly why that is, but clearly file size isn't the only variable.

 

[Mt Spokane Photography]

This is totally incorrect. Photoshop tells you the uncompressed size for RGB DATA!!!!

RAW images store a SINGLE color value per PIXEL. Pixel being directly correlated to the physical element on the sensor. When Photoshop opens an image, it generates an in-memory RGB-per-pixel bitmap. If you loaded the image as 8-bpp, then that is a total of 24-bits per pixel, rather than 14-bits. When you account for the compression, THAT is where you get the 2:1 to 3:1 "ratio".

Same thing goes for saving an image as TIFF. The TIFF image is again RGB, so three color values per pixel. Throw in the increase to 16-bit, and you are radically inflating the actual necessary data sizes here.

I came up with an ~10mb difference by calculating the actual necessary data storage to contain ONE 14-bit data value per sensor pixel, and compared that to actual file sizes. I just looked them up online. As I said, it's very simple math.

Your simple math does not line up with reality. The early test files have doubled in size.

Figures are available already for Nikon models that offer lossy compression, lossless compression, and uncompressed. They also run slightly greater than 2:1.

Scroll down to the section listing typical card capacities where it gives expected file sizes.

http://imaging.nikon.com/lineup/dslr/d4/spec.htm

 

[jrista]

This is totally incorrect. Photoshop tells you the uncompressed size for RGB DATA!!!!

RAW images store a SINGLE color value per PIXEL. Pixel being directly correlated to the physical element on the sensor. When Photoshop opens an image, it generates an in-memory RGB-per-pixel bitmap. If you loaded the image as 8-bpp, then that is a total of 24-bits per pixel, rather than 14-bits. When you account for the compression, THAT is where you get the 2:1 to 3:1 "ratio".

Same thing goes for saving an image as TIFF. The TIFF image is again RGB, so three color values per pixel. Throw in the increase to 16-bit, and you are radically inflating the actual necessary data sizes here.

I came up with an ~10mb difference by calculating the actual necessary data storage to contain ONE 14-bit data value per sensor pixel, and compared that to actual file sizes. I just looked them up online. As I said, it's very simple math.

Your simple math does not line up with reality. The early test files have doubled in size.

Figures are available already for Nikon models that offer lossy compression, lossless compression, and uncompressed. They also run slightly greater than 2:1.

Scroll down to the section listing typical card capacities where it gives expected file sizes.

http://imaging.nikon.com/lineup/dslr/d4/spec.htm

Those file sizes are illogical. This is really basic math. Excluding the masked border pixels, which would increase file size, this is a matter of bits and bytes. The sensor in the D4 is a bayer sensor. That means there is a header in the NEF that defines the bayer pattern, among other bits of metadata, and that is probably only a matter of bytes in size, so a minuscule fraction of the data. The rest of the data is pixel information. Every pixel is stored in a sequence with a specified run length, as a single value of the specified camera precision (in the case of the D4, apparently either 12 bits or 14 bits), until the last pixel is written.

So, ignoring the masked border pixels, the sensor size is 4,928x3,280. Multiplying those together give us the total pixel count of 16,163,840. That is the exact number of light sensitive pixels. To store all of them with UNCOMPRESSED 14-bit precision, mathematically, you need 16,163,840 * 14 BITS, or a total of 226,293,760. Since there are 8 bits per byte, we divide that by 8 to get the number of BYTES, which comes out to 28,286,720. An uncompressed 14-bit RAW image form a sensor with 4,928x3,280 pixels will consume 28.3MB worth of space if it is stored at full precision, uncompressed.

Now, the page you linked says an uncompressed 14-bit NEF for the D4 is 15.3 MB. I'm sorry, but that plain and simply cannot be the case. Something has to give in order for that to be true. Either those files are actually still compressed in some way, or Nikon is actually gimping you on the bit depth...or, the number they are reporting for uncompressed 14-bit RAW is simply wrong. I am not inclined to think Nikon is gimping their photographers in terms of precision, so they are either still compressing those images, or the numbers are wrong. The numbers don't add up in any way that I can account for...so I'm inclined to say the numbers, all of them, for the D4, are just wrong.

I trust the math, more than some spec page on the internet, even if it is official.

I've read around the net that the average image file sizes for the 5Ds range from low 60's to low/mid 70's. With an image size of 8688x5792, it's uncompressed file size would be 88MB. At 62mb, the image would be 70.5% of the full uncompressed size, and at 74mb the image would be 84.1% the full uncompressed size. That is a top savings of less than 30%. Maximum savings would be around 26mb, minimum savings would be around 14mb.

In the case of my 5D III, which is the key source where I actually pulled the ~10mb difference from, my uncompressed size is ~39MB, and my actual sizes in practice range from 27MB to 32MB. At 27MB, I'm saving ~30%, at 32MB I am saving ~18%.

Simple math. Assuming, of course, that were really talking about totally uncompressed, full precision RAW data. God only knows what Nikon is reporting on the D4 page, but it isn't uncompressed full precision RAW file sizes. As for the in-memory load when opening an image in PS, that is the RGB (three full precision values per pixel) load. In the case of a 5Ds, with it's 8688x5792 pixel images, at 8-bpp, the in-memory load should be somewhere around 150,962,688 bytes (~150MB). If we are talking 16-bpp, then that jumps to 301,925,376 bytes (~300MB). If you look around the net, people are seeing 145-150mb loads for 8-bpp and right around 300mb loads for 16-bpp with 5Ds images loaded into Photoshop.

Simple math here.