EOS-1D X Mark II Claims of 15 Stops of DR [CR3]

[StatisticsRule]

I guess we'll see, but I'm so skeptical of Canon these days, between things like the C300 II "15 stops" negative dB measurements,

Out of curiosity, where did you see anything about negative dB?

The best explanation I have seen of what Canon did comes from: https://www.cinema5d.com/canon-measured-15-stops-dynamic-range-c300-mark-ii/

There is little mention of dB, but looking at the plot, the 15th step in the waveform monitor appears to me like it is fairly near SNR=1 (0dB). Of course if I had the data instead of a screen capture, I could be more precise in determining what the SNR of that last step is. In any case, the article by cinema5d claims to be more objective by using IMATEST, but according to the IMATEST website (http://www.imatest.com/docs/dynamic/):

A dynamic range corresponding SNR = 1 (1 f-stop of noise) corresponds to the intent of the definition of ISO Dynamic range in section 6.3 of the ISO noise measurement standard: ISO 15739: Photography — Electronic still-picture imaging — Noise measurements. The Imatest measurement differs in several details from ISO 15739; hence the results cannot be expected to be identical.

Or in other words, the IMATEST measurement used by cinema5d to "disprove" Canon's claim of 15 stops does not conform to ISO standards for dynamic range.

Decisions about "usable" or "useful" DR are subjective and I won't argue that point, but I have yet to see anything inaccurate about Canon's measurements. I am sure you know that SNR=1 (dB=0) means the signal is the same level as the noise and so yes, it will look bad. Where did you see they used negative dB though?

 

[privatebydesign]

I don't understand where the insinuation that Canon are measuring below 0dB comes from, or that they are including any information below that level in their measurements of the C300II DR. They state quite clearly "indeed, the definition of Dynamic Range is when the final lower step has a signal to noise of approximately 0 dB."

Or, everything between FWC and 0dB, the issue seems to be that people (or their testing software) seem to take an arbitrary dB value and say anything below that isn't usable, which would be fair if everybody agreed a single value for 'too much noise' and all sensors behaved the same way at the same dB level, but we know they don't. Besides, even Canon themselves say "Even with the impressive 67dB Luma signal to noise specification this means those two steps ARE noisy." but who are we to judge what is too noisy for somebody else? We can't, some people will use the D5 at 1,000,000 iso and be happy with the results, others will say their usable max is 25,000 iso!

As for Canon not being able to make better sensors, that is farcical considering some of the specialist equipment they actually make, the simple reason they have not made them for general camera release up to now is cost and marketing, simply put they have not been sufficiently pushed by other camera manufacturers (their market share has not suffered enough) to force them to spend the money and absorb the costs. They believe they have not needed to give us 'better' sensors up to this point, but indications seem to be that they are prepared to do that now.

 

[privatebydesign]

Or in other words, the IMATEST measurement used by cinema5d to "disprove" Canon's claim of 15 stops does not conform to ISO standards for dynamic range.

Oh, don't spoil their fun by pointing out one person is measuring to international standards and one isn't. ;D

 

[jrista]

From Canon's own plot:

The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise. The way I believe Cinema5D evaluated that plot is that given the last two stops have no separation between the noise in the signal and the read noise, it's all noise. So SNR would not be >0dB for those last two stops. The 13th stop had a very small amount of separation between the two, so it was the first stop with >0dB SNR.

Given the thickness of the band of shot noise at the top of each signal strip grew with each successive stop, the signal for the last two stops would have been completely buried in noise, hence the reason I called it negative decibels. I don't know what else to call it, not the way I understand this anyway.

Noise is noise. Your going to have, at the very least, shot noise and read noise in every image, assuming zero dark current and a perfect sensor devoid of any defects and identical pixel response. The SNR of a signal cannot ignore the shot noise:

SNR = Signal/SQRT(Signal + ReadNoise^2)

So I understand why Cinema5D has a different interpretation of the plot than Canon, and personally I agree with their interpretation. I expect the 1D X II to be similar...however I'd be pleased if it was something totally new, with a much thinner band of read noise at the bottom of the graph.

 

[Loswr]

From Canon's own plot

There is signal visible above the noise (although not separated) in the 14th and 15th stop, and obviously the scale of the plot is not ideally suited to distinguishing differences under 10.

..the C300 II "15 stops" negative dB measurements...

I'm confused...where is the negative dB on that plot?

 

[privatebydesign]

Well you might not know what to call it, but cinema5D say "Was Canon’s evaluation parameter of 0db as a threshold too loose or are cinema5D’s testing parameters maybe too strict?" so clearly they don't believe Canon were counting sub 0dB stops, and the ISO, by all accounts, also consider Canons methodology to be sound.

As I said earlier, the problem with the way "everybody else" is measuring DR is that it isn't to an international standard and there is no agreement on when 'noisy' is too noisy and shouldn't be counted. However, as we do have an international standard, and Canon use it, I truthfully don't see what the issue is.

Again, as I said earlier, Canon are not saying the C300 II has more DR than another camera, they are just stating what their measurements are when done to that international standard.

 

[privatebydesign]

From Canon's own plot

There is signal visible above the noise (although not separated) in the 14th and 15th stop, and obviously the scale of the plot is not ideally suited to distinguishing differences under 10.

..the C300 II "15 stops" negative dB measurements...

I'm confused...where is the negative dB on that plot?

Obviously jrista considers the bottom of the top white stripe to the top of the bottom white stripe (0dB) be the clear signal, he is saying because the bottom of the top white stripe goes below the top of the bottom white stripe, that he considers 0dB, then stops 14 and 15 are "below 0dB", though nobody else actually agrees with him, including cinema5D , Canon, and the ISO.

Everybody else seems to consider 0dB as the top of the bottom white stripe and the signal to be the top of the top white stripe.

 

[Loswr]

I bviously jrista considers the bottom of the top white stripe to the top of the bottom white stripe (0dB) be the clear signal, he is saying because the bottom of the top white stripe goes below the top of the bottom white stripe, that he considers 0dB, then stops 14 and 15 are "below 0dB", though nobody else actually agrees with him, including cinema5D , Canon, and the ISO.

My bad. I thought zero meant...zero. I mean, it's right there at the bottom of the labels for the ordinate axis. Perhaps further explanation is required? Maybe theoretical physicist Michio Kaku can help with the concept of 'zero'.

http://youtu.be/UV5Wo7YrZIU

;D

 

[privatebydesign]

From Canon's own plot:
...
The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise
....

Interesting that there is almost a straight line down from the 1st stop to the 11th. Makes me want to think that perhaps 12 and beyond are ... pretending.

Is a similar plot available for either Nikon or Sony cameras of interest?

Good god dilbert, surely even you can follow this and not come out with the idiotic "pretending" nonsense?

As for anybody else having similar read outs, it would be great if they did, and there is no doubt that the Arri one would look noticeably better, as would the Sony and Nikon stills cameras when compared to a similar plot from a Canon stills sensor.

 

[jrista]

I bviously jrista considers the bottom of the top white stripe to the top of the bottom white stripe (0dB) be the clear signal, he is saying because the bottom of the top white stripe goes below the top of the bottom white stripe, that he considers 0dB, then stops 14 and 15 are "below 0dB", though nobody else actually agrees with him, including cinema5D , Canon, and the ISO.

My bad. I thought zero meant...zero. I mean, it's right there at the bottom of the labels for the ordinate axis. Perhaps further explanation is required? Maybe theoretical physicist Michio Kaku can help with the concept of 'zero'.

http://youtu.be/UV5Wo7YrZIU

;D

An SNR of 0dB is when the signal strength is equal to the noise, not when the plot is zero (if the plot was at zero, then there wouldn't even be any noise...for that matter, if we go by your interpretation of the graph, where zero on that graph is actually 0dB, then stops 16 and up would also be an SNR > 1...that clearly is not the case, so your interpretation of the 0 line on the graph also being 0dB SNR must be incorrect).

For stops 14 and 15 in the plot, there is nothing but noise...it is white from the bottom of the graph to the top of the peak for that band in the image. I think the scale of the graph is fine, the thickness of the noise in the signal for each band kept getting thicker for each stop darker strip...and the thickness of the noise for the final two stops is much thinner than the thickness for the 13th stop. Given that, I interpret those two strips to have a signal strength less than the noise (total noise, the signal may just barely be stronger than the read noise alone, but again...you cannot ignore the noise intrinsic in the signal itself). The moment that happens:

SNR = 0.99/1 = -0.0873dB

 

[Loswr]

I interpret

Ahh, well then. It seems others interpret differently.

 

[Neutral]

I interpret

Ahh, well then. It seems others interpret differently.

Sounds like a troll statement.
Others is not equal to ALL others )
For anyone who deals with systems noise as part of high sensitivity systems designs/development/deployment it is all clear from the first glance at this graph. Strips 12 and 13 give instant visual representation of SNR and what is approximate SNR for strip 14.
So nothing wrong with the graph scale.
So I do not see here 15 stops DR and tend to agree with JRista.
Will be happy though to see that 1DXII have real 15 stop DR and more than one stop of better high ISO performance.
Also will be happy if Canon will be able to prove that scientifically disclosing to public all the tests methodology )
Personally I do not see any point to argue about specifications which is yet unknown.
This is just waste of time.
I beleive that 1DXII will be definitely better than 1DX but how much better we yet have to see in near future.

 

[JMZawodny]

From Canon's own plot:
...
The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise
....

Interesting that there is almost a straight line down from the 1st stop to the 11th. Makes me want to think that perhaps 12 and beyond are ... pretending.

Is a similar plot available for either Nikon or Sony cameras of interest?

Good god dilbert, surely even you can follow this and not come out with the idiotic "pretending" nonsense?

Stops 1-11 follow an almost linear degradation in signal. Even the ... I don't know what to call it, dark current noise? ... at the bottom of the ISO 100 blip degrades in a similar fashion. Up until the 12th stop. The 12th stop should be lower than it is and 13-15 shouldn't be visible. If you're unsure about what I mean, get a ruler or a piece of paper, put it up on the screen and you'll see that the graph degrades in almost a straight line until 12. Why? What's going on at 12-15?

But there's something else in that graph too that nobody has talked about and that is all of the white dots in the bottom section of the graph from stops 1-5 and is almost gone by 6.

It's not the "wave" that seems to be there at the top.... what gives rise to that behavior?

Is that evidence of Canon's high read noise?

Do you have any idea what that curve is supposed to look like? It does have a well defined theoretical shape. (Hint: all of the points have a small positive bias added to them) It looks just fine to me. Additionally, the fact that there is a bump at the 14th and 15th steps is the significant aspect of this plot, not the lack of separation. Said differently, averaging many of the data would eventually produce a useful (clear) signal. I'd like to see these plots produced for any/all cameras. (unfortunately this appears to be an analog video signal direct from the chip) I'd also like to see the results for the R, G, and B pixels plotted separately as this would tell us a lot about the color accuracy at very low signal.

 

[JMZawodny]

...
Do you have any idea what that curve is supposed to look like? It does have a well defined theoretical shape. (Hint: all of the points have a small positive bias added to them) It looks just fine to me. Additionally, the fact that there is a bump at the 14th and 15th steps is the significant aspect of this plot, not the lack of separation.

It should be a "straight line" through the mid point of all the noise bars.

You can read more here:

http://www.clarkvision.com/articles/digital.sensor.performance.summary/

Incorrect! If you take a strictly power progression plotted on a log plot, it is indeed a straight line. However, if you simply add a small constant to that same data, the line begins to curve as the value approaches the constant and becomes asymptotic to the constant.

 

[StatisticsRule]

The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise. The way I believe Cinema5D evaluated that plot is that given the last two stops have no separation between the noise in the signal and the read noise, it's all noise. So SNR would not be >0dB for those last two stops. The 13th stop had a very small amount of separation between the two, so it was the first stop with >0dB SNR.

jrista, thank you for explaining how you got the negative dB estimate. I have to agree with others that the band separation is not the important point though. To help explain, I created some random data in MATLAB. When the x-axis is between zero and 5, the signal is exactly 0. Between 5 and 10, the signal is exactly 1. On top of that signal, I add 100,000 points of normal random distributed noise with std dev = 1. Finally, to look similar to Canon's plot, I offset all the data by 4 units.

I'd say the data is a reasonable approximation (qualitatively) to Canon's data for the high DR values. Most importantly, because I created the data, I know the right-half of the data has exactly SNR = 1 by definition of the signal height and the noise statistics. You can see the separation between steps is much smaller than the noise height.

From the crude MATLAB approximation here and the screenshot from Canon, I personally still believe Canon shows at least 14+ stops DR. Of course there is only so much you can get from a screenshot.

(As a side note, you might argue I did not use Poisson statistics, but by the time you make the mean value large enough so the data does not show discrete height steps, the poisson distribution is essentially the same as a normal distribution.)

 

[Loswr]

I interpret

Ahh, well then. It seems others interpret differently.

Others is not equal to ALL others )

True. But consider that one of those others is the International Organization for Standardization (see ISO 15739).

 

[Lee Jay]

From Canon's own plot:

The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise. The way I believe Cinema5D evaluated that plot is that given the last two stops have no separation between the noise in the signal and the read noise, it's all noise.

When you shoot a dim nebula, the separation between its light and the skyglow is zero - they overlap. Yet, you are able to see the nebula anyway because its light is added to the skyglow, and you can thus lop-off some of the combined signal.

In other words, the lack of a gap between the shot noise and the read noise floor is not the same thing as "all noise" or zero signal-to-noise ratio.

 

[jrista]

(As a side note, you might argue I did not use Poisson statistics, but by the time you make the mean value large enough so the data does not show discrete height steps, the poisson distribution is essentially the same as a normal distribution.)

First off, I agree with this.

The white band at the top of each strip IS noise. It's the noise intrinsic to the signal itself...the shot noise. The way I believe Cinema5D evaluated that plot is that given the last two stops have no separation between the noise in the signal and the read noise, it's all noise. So SNR would not be >0dB for those last two stops. The 13th stop had a very small amount of separation between the two, so it was the first stop with >0dB SNR.

jrista, thank you for explaining how you got the negative dB estimate. I have to agree with others that the band separation is not the important point though. To help explain, I created some random data in MATLAB. When the x-axis is between zero and 5, the signal is exactly 0. Between 5 and 10, the signal is exactly 1. On top of that signal, I add 100,000 points of normal random distributed noise with std dev = 1. Finally, to look similar to Canon's plot, I offset all the data by 4 units.

I'd say the data is a reasonable approximation (qualitatively) to Canon's data for the high DR values. Most importantly, because I created the data, I know the right-half of the data has exactly SNR = 1 by definition of the signal height and the noise statistics. You can see the separation between steps is much smaller than the noise height.

From the crude MATLAB approximation here and the screenshot from Canon, I personally still believe Canon shows at least 14+ stops DR. Of course there is only so much you can get from a screenshot.

Is your signal just representative of shot noise, or does it also factor in read noise? Based on Canon's plot, it appears that signal starts at the zero line (you can see the bits of shot noise between each band reaching down all the way to the zero line in the graph in a few places, below the band of read noise). I believe it is only the read noise band that is actually offset from 0 by about 4 units. If you remove read noise from the plot, stops 14 and 15 would indeed have separation of the shot noise above the zero line, and the snr would be > 0dB. However with read noise added in, I do not believe that remains the case.

That is not to say there isn't any signal...of course there is signal...it's just that the signal is buried in the noise, SNR <1. If we are talking single-shot imaging, particularly with the CHARACTERISTIC of that noise (see the image below), I don't think those last two stops are very usable by any means (maybe if your doing non-artistic work...police photography of a crime scene or something):

The Canon image has a purple band running right up through stop 12, and a good deal of banding running through stop 13 and up. The Arri, on the other hand, has very clean noise up through stop 18. Those upper stops on the Arri are undoubtedly no better, in terms of SNR, than Canon's stops 14 and 15...they are likely less than SNR 1, however they are much more usable. Similar to an Exmor sensor...you can easily dig very deep into the noise floor and pull out usable information, which is not the case with Canon data (about a seven stop or so lift here, well beyond what should be possible with either camera, but the A7r held up extremely well regardless; top row is just the shadow push, bottom row included additional processing to restore some aesthetic appeal, as much as was possible, to each image):

I also wonder what the power of those bands, particularly that purple band, is in Canon's data. I don't think a simple 2D slice of the signal is sufficient to explain how far those bands may protrude into the signal...and since the purple band is visible at least at stop 12, it's certainly more powerful than the signal at higher stops.

Now, if we are talking astrophotography, we have the option of stacking to reduce noise. Stack 4 subs and you'll reduce the noise by half. That would undoubtedly improve the SNR over 1 in stop 14. Stack 16 subs and it'll probably make stop 15 viable. Stack 64 subs, and your probably good across the board. That isn't going to help much with sports photography or anything like that, though.

 

[StatisticsRule]

Is your signal just representative of shot noise, or does it also factor in read noise?

My goal was not to try and guess the noise source in the images, but rather to come up with an approximation of the signal we are seeing. The calculation for how I did that was described with my last post.

You do raise an interesting point as to what falls under the category of noise, and I will admit I can imagine some cases where the offset could be considered "noise" even though I did not. Offhand, I don't know how the ISO standard defines noise.

If we are talking single-shot imaging, particularly with the CHARACTERISTIC of that noise (see the image below), I don't think those last two stops are very usable by any means (maybe if your doing non-artistic work...police photography of a crime scene or something):

No argument from me. Everyone has their own interpretation of useful signal, and Canon has historically shown banding in the images. Hopefully the 1Dx M II will take care of that as well. My main point was that by everything I see, Canon appears to have a legitimate claim of 15 stops DR.

Anyhow, this has been a good discussion. If nothing else, it has made me re-examine some of my previous assumptions and I learned a few things I had not completely considered before.

 

[jrista]

Is your signal just representative of shot noise, or does it also factor in read noise?

My goal was not to try and guess the noise source in the images, but rather to come up with an approximation of the signal we are seeing. The calculation for how I did that was described with my last post.

You do raise an interesting point as to what falls under the category of noise, and I will admit I can imagine some cases where the offset could be considered "noise" even though I did not. Offhand, I don't know how the ISO standard defines noise.

If we are talking single-shot imaging, particularly with the CHARACTERISTIC of that noise (see the image below), I don't think those last two stops are very usable by any means (maybe if your doing non-artistic work...police photography of a crime scene or something):

No argument from me. Everyone has their own interpretation of useful signal, and Canon has historically shown banding in the images. Hopefully the 1Dx M II will take care of that as well. My main point was that by everything I see, Canon appears to have a legitimate claim of 15 stops DR.

Anyhow, this has been a good discussion. If nothing else, it has made me re-examine some of my previous assumptions and I learned a few things I had not completely considered before.

Same. Your Matlab example makes me wish I had it, as I think it would make demonstrating what I try to explain with just math and formulas easier at times.

Regarding the bias offset, at 14-bit ADUs, it does contain some banding, potentially up to 2 ADU in my astrophotography testing. Removal of the bias signal can sometimes reduce vertical banding in a Canon signal in the deepest reaches of the signal. However if you do so, you have to restore another offset (called a pedestal) that is completely devoid of noise before subtracting the bias. If you do not, you'll clip negative values due to the read noise.

Canon's bias offset is also 512 ADU...at least, it is in their DSLRs. I am a little surprised at how much read noise is indicated by Canon's chart, as despite the offset, it reaches right back down to nearly the zero line. It may be that the C300 II has a smaller bias offset, which might explain the read noise more.