DxOmark's latest king of the hill, dethroning the D800 by some irrelevant points

[Aglet]

DxOmark's new king of the DR hill is a crop video sensor. ouch!

Red's Epic Dragon - so video is not a handicap to excellent sensor performance. I guess I'd based that tenet on some other companys' products.

www.dxomark.com/Reviews/RED-Epic-Dragon-review-First-camera-to-break-the-100-point-DxOMark-sensor-score-barrier

Altho not very flexible as a still camera, just goes to prove that well designed hardware can produce excellent performance metrics even at video framerates, ... with a crop sensor. I'm impressed, too bad we don't get to view the basic measurement data.

I wonder how that Arri would test.

 

[Mt Spokane Photography]

Since when does DXO feel the need to justify their ratings by actually testing a camera. They measure a sensor and come up with outlandish results and give a camera a rating based on the sensor test alone.

Having had a D800, I can say its a nice camera, but overall, it fails to match my 5D MK III. That's why they are being dumped by buyers. They need someone who is very careful and knows what they are doing to get benefit of the sensor, and that amounts to only a small percentage of buyers.

The Red is going to require even more expertise, and a lot of people are going to nail DXO for their lack of transparency in testing and rating. They don't care, they march to the beat of a different drummer.

 

[Aglet]

Well, without them coming out about it bluntly, the no-analog-gain Dragon sensor shows FPN by the time it's digitally pushed to 3200, no surprise that.

As for the D800, it is my favorite and most capable landscape camera. Overkill for many other tasks but I enjoy shooting with the quirky things. It's certainly not a jack-of-all-trades nor a camera for the masses. I won't be trading in mine for anything on the horizon, not even the moderately priced new Hassie or 645D2... Well, the latter might be tempting.

 

[Loswr]

http://www.dxomark.com/About/Sensor-scores/Overall-Score

what do you think is weird with DXO measurements?

Their measurements are generally useful (when correct - their lens measurements have some significant errors). It's their Scores, to which you linked, that are problematic...

• They don't disclose how they compute the 'average'
• Two of the three subscores consider only ISO 100, but not everyone shoots at ISO 100 all the time
• Their 'Sports Score' is a total misnomer - low noise at high ISO coupled with poor AF or a slow frame rate is not good for Sports
• The Score for DR can exceed the maximum possible electronic/physical range, such as 14.4 stops of DR for a sensor with a 14-bit ADC.

I could go on, but there's little point as I've said it all before. DxOMark's scores are biased, and Biased Scores = BS.

 

[preppyak]

• Two of the three subscores consider only ISO 100, but not everyone shoots at ISO 100 all the time
• Their 'Sports Score' is a total misnomer - low noise at high ISO coupled with poor AF or a slow frame rate is not good for Sports

I'll actually disagree, not with it being misnamed, but with it being irrelevant. Again, they are measuring only the sensor, so its probably quite true that in a body with a fast frame rate and good AF, the D800 sensor would be superior. That body just happens to not exist from Nikon, which is irrelevant for what DxO concerns itself with.

But you are right about "sports" being the wrong name. If they just called it their low-light score, then people would just be complaining about what type of noise they prefer. Especially Street or photojournalism would be more applicable than sports.

Because obviously a camera with that DR, decent ISO handling, shooting RAW, and the ability to shoot 60 fps at resolutions higher than what Canon and Nikons top end cameras do, is WAY better for sports in the theoretical.

 

[Loswr]

I quote Prof Eric Fossum, the active cmos sensor's father, http://ericfossum.com/Publications/Pub%20List.htm
http://www.ericfossum.com/Patents/Full%20Patent%20List.htm
and about DXO sensor scoring
it is always good when you score high on a standardized objective test

I think it says it all

Sorry, but it says nothing. In the US, there is a standardized test called the SAT, used for college admission (it used to be called the Scholastic Aptitude Test, but they dropped the full name after it became widely accepted that the test does not predict aptitude in any meaningful way). There are lots of people who score highly on that test and perform very poorly in college.

In other words, scoring high on a standardized test simply means that you have the ability to take the test well. It says absolutely nothing about your ability to perform in the real world. Similarly, scoring high on a DxOMark test says nothing about the ability of a camera to deliver good image quality.

The fact that he's the 'father of the active COMS sensor' doesn't make Fossum the ultimate authority on the subject, any more than DxO. Linus Pauling, who was awarded the Nobel Prize in Chemistry, proposed the structure of DNA as a triple helix…oops.

 

[mackguyver]

I quote Prof Eric Fossum, the active cmos sensor's father, http://ericfossum.com/Publications/Pub%20List.htm
http://www.ericfossum.com/Patents/Full%20Patent%20List.htm
and about DXO sensor scoring
it is always good when you score high on a standardized objective test

I think it says it all

Like Neuro, I'm confused. First of all, where's the quote? Whether English is your first language or not, the words generally go in-between two " symbols. Second, while the links show an impressive list of academic publications and patents, what do they have to do with this discussion? Third, relating back to the first, what says it all? If I'm reading your post correctly, Dr. Fossum said it's good to score high on standardized tests, but what does that have to do with DxO?

I'm not trying to be rude, but I'd like to understand your argument and why you, "Think it says it all." I'm in agreement with Neuro on DxO - their measurements are generally good, I really like their software, the DxOMark site is free, so I can't complain, but their "scores" seem to have little to do with reality and they should do a better job disclosing their methods.

 

[Loswr]

Now I do not understand what you mean, Eric has commented Reds and DXO measurement in another forum with the words, it is always good When You score high on a standardized test Objective. Its about FWC, QE, read out noise ,DR
And its about the Reds sensor, not AF or similar

I believe the confusion comes from the fact that it is accepted practice when you quote someone and provide a link, that the link is to the actual quote which you excerpted (demonstrating both the validity of the quote, and the context in which it was made), and not a bibliography of that individual it is neither useful nor relevant in the present context.

Any standardized test only has meaning if the scores resulting from the test itself are both unbiased and relevant to real world performance. DxOMark's scores are neither.

 

[sagittariansrock]

Now I do not understand what you mean, Eric has commented Reds and DXO measurement in another forum with the words, it is always good When You score high on a standardized test Objective. Its about FWC, QE, read out noise ,DR
And its about the Reds sensor, not AF or similar

And about Eric Fossum, it's always good to explain that it is one of the world's leading sensor experts who speaks out and no clown about DxOmark's latest king of the hill, dethroning the D800

Can it be that Red and Sony have the highest scores ​​and Canon sensors much lower
and DXO data therefore should not be seen as reliable?

You should quote completely and accurately:

Eric Fossum said: "The Red camera designers should be congratulated on the high performance. I am not a big fan of the DxO scoring system but it is always good when you score high on a standardized objective test. Congratulations especially to the sensor design team. Very happy for you guys. You know who you are."
http://image-sensors-world.blogspot.com/2014/03/red-epic-dragon-sensor-wins-dxomark.html?showComment=1394016545431#c2295589572615330146

Got it? The leading sensor expert, while congratulating what is most likely an excellent "sensor" (not necessarily the camera, by the way, which most people here care about) also expressed his reservations about DxO scoring system.

 

[Loswr]

Now I do not understand what you mean, Eric has commented Reds and DXO measurement in another forum with the words, it is always good When You score high on a standardized test Objective. Its about FWC, QE, read out noise ,DR
And its about the Reds sensor, not AF or similar

And about Eric Fossum, it's always good to explain that it is one of the world's leading sensor experts who speaks out and no clown about DxOmark's latest king of the hill, dethroning the D800

Can it be that Red and Sony have the highest scores ​​and Canon sensors much lower
and DXO data therefore should not be seen as reliable?

You should quote completely and accurately:

Eric Fossum said: "The Red camera designers should be congratulated on the high performance. I am not a big fan of the DxO scoring system but it is always good when you score high on a standardized objective test. Congratulations especially to the sensor design team. Very happy for you guys. You know who you are."
http://image-sensors-world.blogspot.com/2014/03/red-epic-dragon-sensor-wins-dxomark.html?showComment=1394016545431#c2295589572615330146

Got it? The leading sensor expert, while congratulating what is most likely an excellent "sensor" (not necessarily the camera, by the way, which most people here care about) also expressed his reservations about DxO scoring system.

Thanks for posting the full quote and proper citation.

I quote Prof Eric Fossum...

You quote someone who knows a lot more about sensor design than you, but you quote him inappropriately and out of context.

It's obvious that you intentionally posted the quote out of context, and lacking the reference. It's equally obvious that you posted the link to Fossum's bibliography to engender additional respect for the 'father of the active CMOS sensor'. Frankly, it makes you look rather foolish, or at best disingenuous, when 'one of the world's leading sensor experts' states that he is, "...not a big fan of the DxO scoring system," an opinion with which I resoundingly agree.

I think it says it all

I think that says it all about your credibility - you have none.

 

[philmoz]

We seem to have different ways of interpreting the words: but it is always good when you score high on a standardized objective test , and the negative about DXO is not about the meassured sensor criteria as DR, FWC ,QE , color resolution, low light and so on

Which would be true if DxO was an objective test.

Phil.

 

[mackguyver]

sagittariansrock, thank you for the actual quote, and I'm happy to see that the Professor is on the same page with most of us.

philmoz, I have no reason to believe that DxO's measurements aren't objective. If you've read or seen much about them, they seem very exacting about their measurements. The "Scores" are also objective, assuming they are consistent and based on the measurements. The criteria used to calculate the scores is subjective, however, and I think that's what we don't like. While they are free to come up with any formula they want, it would be nice if they were clear about what that formula was, and without that clarity all we can do is speculate. While I think most of us would agree that the measurements translate to real-world performance, the "scores" do not.

Nalle Puh, it's okay to be passionate about a subject (and to disagree with others), but if you're going to quote people, please do it responsibly and accurately. Unfounded arguments aren't going to get you very far on a forum full of people who like facts and accurate information.

 

[Loswr]

Can it be that Red and Sony have the highest scores ​​and Canon sensors much lower
and DXO data therefore should not be seen as reliable?

DxOMark Scores ≠ DxoMark data. Can you explain exactly how their measurement data are used to generate the scores? No, because they don't publish that information. In the case of lenses, the measured data on optical performance have almost nothing to do with the Scores.

As I've stated many times, their measurements (= data) are useful, although occasionally wrong, but their Scores are biased and useless, i.e. not reliable.

Several peoples approach to DXO seems to be negative as long as the Canon does not get high sensor scores

Perhaps, but speaking for myself, my opinion of DxO's Scores is negative because they are biased and determined in an unknown manner. While their software is excellent and their measurements are generally well done and useful, they are not perfect. Their data are sometimes flat out wrong, e.g. their conclusion that the original 70-200/2.8L IS is better than the 70-200/2.8L IS II. More damning to their credibility is the way they handled their error - they denied that there was any mistake, but subsequently and silently modified their data to reflect the fact that the MkII version of the lens is better.

The honest thing to do when you make a mistake - whether in a lens measurement or posting a quotation out of context - is to own up to it, and correct it openly. A person or a company that denies wrongdoing and obfuscates the issue does not deserve respect.

 

[mackguyver]

http://www.dxomark.com/Reviews/DxOMark-Camera-Sensor2/DxOMark-Score

Don't you mean this one?

Either way, it doesn't give you anything other than the general criteria for each "score", and it still doesn't explain their calculations/formula for how they determine the numerical value of the "score".

 

[Loswr]

http://www.dxomark.com/Reviews/DxOMark-Camera-Sensor2/DxOMark-Score

Don't you mean this one?

Either way, it doesn't give you anything other than the general criteria for each "score", and it still doesn't explain their calculations/formula for how they determine the numerical value of the "score".

As I pointed out in another thread (from which I'll just paste the following), the article's author, Peter van den Hamer, goes on to describe some of the problems with DxO's sensor analyses, such as the low ISO bias of the sensor score (one reason I call them Biased Scores = BS), the fact that measuring color depth (i.e. chroma noise) at low ISO is basically meaningless (and yet it's a major factor in the Sensor Score), their confusing nomenclature for the subscores (e.g. Sports Score), etc.

He also takes issue with DxO's refusal to divulge the way they calculate the overall score. He has come up with an approximation which he suggests is usually to accurate to within 1-2 points: DxOMark_Sensor_Score = 59 + 4.3*(ColorDepth-21.1) + 3.4*(DynamicRange-11.3) + 4.4*log2(ISO/663) -0.2. He also states, "My guess is that the actual formula is non-linear and may use (under some conditions) coefficients of 5/5/5 rather than 4.3/3.4/4.4." His suggestion that the 'master formula' which DxO uses may be modified under some conditions further supports the claim that DxO's scoring is biased.