5DS scores at DXO **now posted**

[Don Haines]

...
It's not the individual numbers, it's how they are put together to get an aggregate score. DXO would be better off if they just left things as the numbers for particular aspects....

You can not come up with an aggregate number without introducing bias. If I feel that colour depth is more important for my style of photography and you fell that DR is best for your style, we will agree that all the measurements are good, but never agree on how the aggregate score is calculated. This is the flaw in DXO... they went beyond taking measurements and introduced bias.....

Can you demonstrate with actual scores for different cameras where you see evidence of this bias?

Dilbert, I don't think that's what he means. In this case, it's not a brand bias so much as 'all photographers all over the world would only want this kind of camera' bias. How they roll up the aggregate score weights things in a way not everyone might want.

A concert photog is probably far far far more concerned about high ISO performance than how much DR the rig has at ISO 100. A studio portraiture person might more heavily prioritize color. A landscaper, on the other hand, might love DXO's aggregate score as it is. Everyone's needs are different.

- A

Dilbert, you are thinking details and I am thinking concept. The details are correct. The concept is flawed.

The very nature of assigning weight to various aspects of camera performance introduces bias. If your needs align with the scoring criteria, it will be helpful. If your needs conflict with the scoring criteria, it will not be helpful. Due to the nature of photography, there are a lot of divergent paths to follow and that means that for most people, the aggregate score is either useless or counterproductive.

There is no way to come up with an aggregate number that will serve all disciplines. Forget the Canon/Sony/Nikon bickering... even if you stay with one manufacturer, it does not work.

 

[jthomson]

I would like to understand the low iso scores of the 5DS/5DSR. My understanding was that the larger pixels of the 5DIII and 6D were what gave them the low light capability. The 5DS/5DSR have smaller pixels yet get similar iso scores to the 5DIII and 6D. How are they achieving roughly twice the iso score of the 7DII with a similar pixel size?

 

[ahsanford]

I would like to understand the low iso light / high iso scores of the 5DS/5DSR. My understanding was that the larger pixels of the 5DIII and 6D were what gave them the low light capability. The 5DS/5DSR have smaller pixels yet get similar iso scores to the 5DIII and 6D. How are they achieving roughly twice the iso score of the 7DII with a similar pixel size?

Great question. See edit I made above. Otherwise I was getting confused with your question versus the other parts of it.

If my edit is what you meant, yeah, we were expecting 7D2 levels of noise at higher ISO with this sensor based on Maeda-san's comments after the 5DS rigs were announced. I was bracing for a step back in high ISO performance compared to the 5D3 for sure.

But TDP and now DXO are saying the 5DS is on level with the 5D3 for high ISO. Just speculating, here are two possible reasons:

1) Is it possible DXO doesn't test SNR at the pixel-level? TDP found that the 5D3 does outperform the 5DS in higher ISO at a pixel level, but when you downsized the 5DS shots, it tempered the noise to the point where they thought the 5D3 and 5DS were on level terms. As for DXO, I've seen no specifics in their Sports/ISO method, but perhaps they also downsize their shots for their ISO/Sports like they do for dynamic range? That might explain how the 5DS fared so well vs. the 5D3.

2) Is the explanation in the processing power? Two DIGIC 6 chips in the 5DS should be able to pull-off more than than one DIGIC 5+ in the 5D3, but I don't know if that's for handling noise nearly as much as those 50 MP files.

I am admittedly not a sensor-whiz. Can anyone with a EE background or boatloads of technical experience on this front help us out?

- A

 

[RGF]

You don't need scores to show that there is bias, you need only their description of the scores.

...the Sensor Overall Score describes the results of measurements only on sensors and is essentially related to image noise (for example, a difference of one f-stop offsets the Overall Sensor Score by approximately 15 points)...

Image noise has a more significant impact on sensor score than color depth, for example. That's called bias. It's perfectly fair for them to set up their own biases (another word could be 'weightings'), I just wish they'd disclose them.

How is DxO ONE SuperRAW™ getting its own separate score a 'measurement only on sensors'?? It's perfectly reasonable for them as a private company to test whatever/however they want, but it ruins their credibility as independent/impartial testers.

Nothing like the fox watching the hen house

 

[Orangutan]

This is the flaw in DXO... they went beyond taking measurements and introduced bias.....

Can you demonstrate with actual scores for different cameras where you see evidence of this bias?

If this were an academic publication, it would be DxO's obligation to reveal their methods so we can determine whether there is bias. The fact that they don't do that is suspicious.

 

[9VIII]

The problem is that the bias against Canon is intentional, they can't "fix" something when it's functioning just as they intend (as smear campaign and propaganda).

In the past, I've defended DxO against claims of direct brand bias. Pretty disappointing to see the post today showing DxO comparing the 'Professional' D810 to the 'Semi-professional' 5DIII.

Maybe if Aglet asks them nicely, they'll fix that mistake, too. I guess he didn't notice them staunchly defending their initial 70-200/2.8 L IS vs MkII results that they changed a year later, probably he was too busy cursing at his inability to use his 5DII.

Does it really matter whether they hiss words of seething hate or are just willfully ignorant?
The forgotten stepchild is still subject to abusive behavior, even though not beaten with a stick.
How long has DxO actually been around? It looks like they didn't start writing reviews until 2008, with the D3x already being the ultimate camera (so good that Canon has yet to "equal" its level of BS).
It's entirely possible that they set out from the very beginning to use a scoring system with two useless metrics that Nikon had already demonstrated they were going to continually inflate.
It doesn't even need to be a conspiracy, if they're a bunch of butthurt Nikon fanboys (all Nikon fanboys are inherently butthurt after watching Canon beat Nikon to every technological advancement that mattered in the past 25 years) then they would be just as eager to do this as continually dropping "Canon sucks" leaflets across the sky.

Canon should go ahead and start an "independent laboratory" that reviews cameras based on image quality at ISO 6400, buffer performance, and the ability to adjust aperture in live view.

 

[9VIII]

http://www.dxomark.com/Reviews/Canon-5DS-5DS-R-Review-New-top-ranking-Canon-EOS-sensor/Canon-5DS-5DS-R-Comparison-3-EOS-5DS-vs-Nikon-D810-vs-Sony-A7R

The Nikon D810 and the Sony A7R, for example, both offer around two-thirds of a stop better image quality

Some people who support DxO try to say that you should just ignore the overall score and "dig deeper". Obviously DxO doesn't feel that way.

 

[Maximilian]

...
There's no evidence to show that the numbers are wrong and there's no evidence of bias - if there was any evidence of bias then it would be trivial ...

Really?
Okay. I'll leave the field of arguing up to you...
Again: I am not saying that Canon sensors are equal to EXMOR sensors or else.
I only say that puting DxO values into argumentation makes the one using them look foolish.
Because they are scientifically wrong. Especially because the method is not laid open.

If you still feel to be right, although DxO have discredited themselves and if you want to continue to defend them just because their numbers are fitting into your arumentation, then okay, so be it.
Makes you look like having a no arument at all in my eyes. But of course you'll see it different. So be it.

So jump up and down and shout all you like about the numbers being wrong or biased... stop acting like a spoiled child.

Funny. To me this looks like you're describing the look into your own mirror. But so be it.
Gn8 mister always right.
: : :

 

[3kramd5]

You don't need scores to show that there is bias, you need only their description of the scores.

...the Sensor Overall Score describes the results of measurements only on sensors and is essentially related to image noise (for example, a difference of one f-stop offsets the Overall Sensor Score by approximately 15 points)...

Image noise has a more significant impact on sensor score than color depth, for example. That's called bias. It's perfectly fair for them to set up their own biases (another word could be 'weightings'), I just wish they'd disclose them.

How is DxO ONE SuperRAW™ getting its own separate score a 'measurement only on sensors'?? It's perfectly reasonable for them as a private company to test whatever/however they want, but it ruins their credibility as independent/impartial testers.

I agree completely. Calling it raw so that they can present it relative to single exposures is indefensible

 

[topdog]

the only thing that matters will be the results of real life photo comparisons and experiences. no amount of arguing or debating is gonna settle anything

 

[Loswr]

the only thing that matters will be the results of real life photo comparisons and experiences. no amount of arguing or debating is gonna settle anything

Yes it will.

 

[Loswr]

the only thing that matters will be the results of real life photo comparisons and experiences. no amount of arguing or debating is gonna settle anything

Yes it will.

No it won't.

 

[ahsanford]

the only thing that matters will be the results of real life photo comparisons and experiences. no amount of arguing or debating is gonna settle anything

Yes it will.

No it won't.

This is how you get 16k+ posts, people.

That's just the sort of laser-guided rightness we need.

- A

 

[Jack Douglas]

Let's face it many people just like arguing. Sadly that also applies outside this forum. I'm looking for photos not arguments. I've read about 5 posts in this thread and that's enough for me, I'm un-checking it.

Jack

 

[Don Haines]

Let's face it many people just like arguing. Sadly that also applies outside this forum. I'm looking for photos not arguments. I've read about 5 posts in this thread and that's enough for me, I'm un-checking it.

Jack

Fortunately, we have a film which covers how to post on this forum......

https://www.youtube.com/watch?v=kQFKtI6gn9Y

 

[PureClassA]

Classic....

Let's face it many people just like arguing. Sadly that also applies outside this forum. I'm looking for photos not arguments. I've read about 5 posts in this thread and that's enough for me, I'm un-checking it.

Jack

Fortunately, we have a film which covers how to post on this forum......

https://www.youtube.com/watch?v=kQFKtI6gn9Y

 

[Orangutan]

This is the flaw in DXO... they went beyond taking measurements and introduced bias.....

Can you demonstrate with actual scores for different cameras where you see evidence of this bias?

If this were an academic publication, it would be DxO's obligation to reveal their methods so we can determine whether there is bias. The fact that they don't do that is suspicious.

But it's not an academic publication

Many engineering and science-oriented companies strive for that same level of integrity. The fact that DxO is not willing to do so as well is a strong indicator of their lack of confidence in their methods.

 

[ajfotofilmagem]

This is the flaw in DXO... they went beyond taking measurements and introduced bias.....

Can you demonstrate with actual scores for different cameras where you see evidence of this bias?

If this were an academic publication, it would be DxO's obligation to reveal their methods so we can determine whether there is bias. The fact that they don't do that is suspicious.

But it's not an academic publication

Many engineering and science-oriented companies strive for that same level of integrity. The fact that DxO is not willing to do so as well is a strong indicator of their lack of confidence in their methods.

What do they gain by doing so, hmm?
They shift the debate in one small corner of the websphere from X to Y...
Do you really think people here will stop hating on DxO if they published their formula?

Will the people will stop hating a corrupt politician, because he has made public its methods to steal the money of the country he governs?

 

[Don Haines]

It's not the individual numbers, it's how they are put together to get an aggregate score. DXO would be better off if they just left things as the numbers for particular aspects....

+1. I've been saying this for some time. I don't really have an axe to grind with their metrics (save perhaps their perceptual MPix with lenses), I just can't stand their aggregate scores which categorically disregard the varying needs of photographers. Some folks live in a high ISO world while others live in the studio or on a tripod. It stands to reason they'd have different needs with their gear.

DXO would be a far more respected site if they had the following operating plan:

1) Collect data.
2) Describe your methods.
3) Report data on just the individual metrics.
4) [ crickets ]
...

Well the problem is that you can't prove that (4) would happen any more than I can disprove it

I'm pretty sure that if DxO published the formula for their sensor score that the forums would then erupt into debate as to whether that was correct and that if they changed this or that then scores would be different... or how DxO doesn't take this or that into consideration.

Really, the score is just being used as a distraction by a few people to turn the argument away from Canon scoring poorly (or rather not as well as other manufacturers.)

You aren't getting it....

It doesn't matter what the formula is. It's the fact that they move from recording and reporting all the sub-scores (which is good), to creating a formula for overall ranking....

It does not matter what the formula is. Having it is a bad idea and does a dis-service to everyone.

Put a landscape photographer, a wedding photographer, a portrait photographer, a sports photographer, a news photographer, a bird photographer, a studio photographer, and a cat photographer in the same room and those 8 people will come up with 8 different ways to rank which one is the best camera.

A fair rating system and a universally accepted single rating number are mutually exclusive because different needs require different weights on different metrics. IT CAN'T BE DONE!

 

[benperrin]

the only thing that matters will be the results of real life photo comparisons and experiences. no amount of arguing or debating is gonna settle anything

That's the thing. I know that people here generally dislike Tony Northrup but he already did a comparison test between the 2 cameras and found the 5ds to be better. According to dxomark the d810 should be a much better camera. It simply isn't true. And at least with TN he posts his methods for coming to his conclusions and even tells people that if there was an error with his methodology, to tell him the correct way to do it and he'll re-do the test.