DxOMark vs. Reality

[elflord]

I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

That's true, but it's not the same thing as saying that there do not exist individuals for whom it's useful.  The user for whom such a blunt instrument is useful is not even going to go to the trouble of addressing whether or not the score addresses their use case(s).

It would be better if the aggregate scores were disclosed, though if they can be reverse engineered within a point or two I don't see this as a major issue (again consumers of this aggregate score would not pay attention to the formula if it were published anyway)

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not.

I don't see how coming up with a benchmark for every conceivable use case (or just not publishing use case benchmarks at all, thus forcing everyone to go through the measurements)  would make things any more clear or less confusing.

So it's true that some use cases might require more attention to detail on part of the user. The good news is that DxO make it all available.

[elflord]

I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

Photographers "should" pay attention to the actual measurements and properly understand sensor benchmarking.

However, not all "buyers of cameras" are serious photographers, and not all of them are as attentive to the nuances of sensor benchmarking as photographers should be.

Reviewing the raw measurements is better than reviewing the use case scores, which is better than reviewing the aggregate score, which is better than not reviewing any sensor benchmark.

[Zlatko]

I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

I think photographers should look at the measurements when they measure something of relevance to the photographer, and ignore the Overall Score.

[Kernuak]

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.

[leolol]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.

[Kernuak]

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.

I try to follow you but do not understand what you mean.
please explain

In low light conditions, the dynamic range in a scene is much less than any modern sensor can capture, therefore whether or not one sensor can capture more DR than another is irrelevant. As I've never measured it, it is rather subjective, but I would estimate that DR is in the 6 stop range for many of the low light conditions that pushed me into buying the 5D MkIII (that's purely an example and wasn't the only reason for buying it). When you start getting into that sort of lighting, then a test should be looking more at the amount of noise in an image and less at how much DR the sensor can capture. DR is desirable for some photography, but unimportant in others and any test of a sensor should reflect that in any objective testing. What I would like to know, is how DxO come up with their cut-off points. For example, why 9 stops of DR for low light testing? Why not 10 or 8? Like Neuro and one or two others, I have been involved in scientific research (albeit nothing to do with optics) and I know that statistics can be used to emphasise pretty much anything if you don't look too closely. That's why it is important to do the right tests and the right analysis. You get it wrong and don't apply proper rigor to the testing in medical research and you could put patients' lives at risk further down the line.
Ultimately though, photography is an art form and therefore subjective. What one person might find acceptable, someone else might find it objectionable. Nowhere is this more obvious than noise and some even add noise to add atmosphere. Even blown highlights or blocked out shadows can add to an image if done correctly, although again, some people will hate it while others love it.

[Kernuak]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.

Actually, I don't think he was saying any camera or sensor was better than another, rather that the perception of "better" (however it is measured) is actually irrelevant. The only thing that matters for a business is sales figures. If sales or profits are increasing, then they are achieving their goals (at least in most cases).

[neuroanatomist]

@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

No, I base my conclusions on which camera is better (for me) using criteria and performance aspects that matter to me.  In my opinion, that's what everyone buying a dSLR should use to make their own decision, and in most cases, that's likely the case.

It is a fact that Canon sells more dSLRs than Nikon.

Takes together, that suggests that in aggregate, more people have judged that Canon dSLRs satisfy more of their decision-making criteria.

Consider - there's an election, at the end of which (hanging chads notwithstanding), there's a winning candidate and a losing candidate. 'Winner' is based on a count of votes.  You voted for the candidate you thought was 'the best'...but that candidate may not have won.  You're entitled to your opinion, and if you picked the losing candidate, that doesn't invalidate your opinion. But at the end of the counting, you need to understand that the candidate you liked best lost, and your opinion is in the minority.

There are lots of ways to judge something.  The problem is that the vast majority of them are biased by personal opinion.  What's best for you isn't necessarily best for everyone. I use sales figures as a metric because that's one of very few unbiased metrics available.  We could argue all day about which candidate would be 'best for the country' or which camera is 'the best for taking pictures'.  But there was an election, with a winner and a loser...and Canon has sold more cameras than Nikon.

Those are the facts, plain and simple.  If it helps, while you don't have the option to be governed/represented by the losing candidate, you can go buy a Nikon dSLR.

[tortilla]

A skilled photographer is not the same as  he or she  is a good  copyist and know how a digital work flou works.  Have you  anytime worked in a darkroom with a negative film  then you will understand what I mean and likewise it is with a raw files.

A skilled photographer with also skills in the digital "dark room" kan do so much more than a photographer with no  photoshops skills.  And even more with a camera who has 14 stops DR.
best regards , pro since 30 years back.

Well, when you need 14 stops DR to "rescue" your pictures via Photoshop, then you should ask yourself, if you`re a real pro.
When it comes to the the point where you really need 14 stops of DR, you've taken your picture in a wrong way.

And: none will see the difference. Neither a customer nor other pro photographer.

So, does that mean HDR imaging is only for amateurs like me who are not able to exposure right, because they don't find the full-automatic setting?   Real world lighting situations can sometimes have way above 14 EV DR.

[dilbert]

...
Consider - there's an election, at the end of which (hanging chads notwithstanding), there's a winning candidate and a losing candidate. 'Winner' is based on a count of votes.  You voted for the candidate you thought was 'the best'...but that candidate may not have won.  You're entitled to your opinion, and if you picked the losing candidate, that doesn't invalidate your opinion. But at the end of the counting, you need to understand that the candidate you liked best lost, and your opinion is in the minority.

Note that in elections, the outcome of who wins can be more complex where a person can win without having a overall majority of people vote in their favour.  Elections are just another analogy that fails as you could just as easily argue that it is not always the best person or the person with the best policies that wins but rather the person that sells themselves and their policies the best, meaning that a good percentage of people had the wool pulled over their eyes (i.e the winner should not have won.)

But really, lets not drag politics into discussion about cameras.

But there was an election, with a winner and a loser...and Canon has sold more cameras than Nikon.

I prefer to look at trends rather than specific points in time, i.e I'm more interested in knowing if Canon/Nikon's DSLR share is growing/shrinking than what specific percentage it is.

[neuroanatomist]

I prefer to look at trends rather than specific points in time, i.e I'm more interested in knowing if Canon/Nikon's DSLR share is growing/shrinking than what specific percentage it is.

That's fine, as long as you realize that one quarter does not equal a trend...  FWIW, Canon has gained market share at Nikon's expense for the past 5 years or so. 

[dilbert]

I prefer to look at trends rather than specific points in time, i.e I'm more interested in knowing if Canon/Nikon's DSLR share is growing/shrinking than what specific percentage it is.

That's fine, as long as you realize that one quarter does not equal a trend...  FWIW, Canon has gained market share at Nikon's expense for the past 5 years or so.

Well it is hard to see Canon maintaining its sales numbers without continued fire sales of (for example) the 5D3 at a discount of over $500 of the MSRP. Or one might say that such fire sales are evidence that vendors are struggling to sell 5D3 stock at MSRP and that Canon sales numbers will suffer if they don't happen.

[neuroanatomist]

Well it is hard to see Canon maintaining its sales numbers without continued fire sales of (for example) the 5D3 at a discount of over $500 of the MSRP. Or one might say that such fire sales are evidence that vendors are struggling to sell 5D3 stock at MSRP and that Canon sales numbers will suffer if they don't happen.

Fire sale...hyperbolize much?  As for sales numbers 'suffering' I just had a look at Amazon's dSLR sales rankings this busy holiday shopping season, and the top three best sellers are from....Canon (top four on the list, but the Canon T4i is #1 and #4, with different lenses).  And while we all know the D800 is a much better camera (so telleth us the almighty DxOMark), and it's cheaper, too, the 5DIII is in the Top 20 (#14), and the D800 is not (#21).  Canon does seem to be suffering badly, oh my. 

[neuroanatomist]

Is Amazon a reference?

It's a reference in the sense that one can see the data, so claims can be substantiated.  Do you have data that you can show to substantiate the claim than the D800 is outselling the 5DIII 20:1 in Sweden? 

I wonder how many cameras are sold in Sweden, compared to the USA?  There are ~33 people in the US for every 1 person in Sweden.  Raw numbers matter far more than ratios.

Nikon V1 best selling camera in Europe

I have read (in a Nikon press release) that the Nikon 1 was identified as the best selling compact camera system in Europe in 2012.  Where was it published that the Nikon V1 was the best selling camera in Europe?

Nikon has increased  theres SLR selling with 26% 2012.

Not exactly.  I think you're looking at the results for the 1st half of their FY2013, which are the 2nd and 3rd quarters of the 2012 calendar year (Canon and Nikon's fiscal years are offset).  Performance in the first calendar quarter of the year were less impressive, barely making up for the loss in the last half of 2011. 

Over those same two quarters (2Q12 and 3Q12), Canon reported a gain of 47% and a loss of 7%, respectively, in the dSLR segment.  Which is better, when looked at over a 6 month period - a 26% gain, or a 47% gain for the first 3 months and a 7% loss for the next 3 months?

[verysimplejason]

Here's some statistics from flickr.   I don't know how relevant this is but this shows the most used cameras by flickr users.

http://www.flickr.com/cameras/