DxOMark vs. Reality

D

Duprant

Guest
PackLight said:
Not everyone researches as in depth as those on Canon Rumors. I know many with DSLR's that wouldn't have a clue how to interpret DxO's information, and they would just take the overall number at face value. What DxO's motive is I wouldn't know, but when ever a public company offers information there is usually a motive and it almost always involves monetary gain. Why do they post this information at all? Really I don't care.

Their motive is to publicize the DxO brand and sell DxO software. Nothing nefarious here.
 
P

PackLight

Guest
Duprant said:
Their motive is to publicize the DxO brand and sell DxO software. Nothing nefarious here.

What?? You don't think Nikon sent a can of Christmas popcorn or a care package over to DxO this year for all of their help?
 

elflord

EOS RP
Aug 2, 2011
692
0
neuroanatomist said:
Perhaps one more refinement: irrelevant to camera sensor image quality. A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Well if your point is that the final product depends on aggregate performance of the whole system, that's fine, but what are they to do about it ?

They could aggregate AF performance into an even more subjective and debatable score, but you have repeatedly said that this is precisely what they shouldn't be doing.

Are you simply opposed to them publishing any sensor benchmarks because they expose the technological stagnation of your favourite camera brand ?
 

TheSuede

EOS M6 Mark II
Oct 9, 2012
54
0
neuroanatomist said:
Perhaps one more refinement: irrelevant to camera sensor image quality. A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Hehe... Yup - true.

But when you can 'assume with reasonable confidence' that two cameras have similar AF accuracy and sensitivity, the static image quality metric still rules.
-And you might have noticed that I deliberately MENTIONED the autofocus as being left out of the metric in the first place - so no modification is needed.

Better image quality = better image quality.
No modifier, no disclaimer, no BS sidetracking.
 

neuroanatomist

I post too Much on Here!!
CR Pro
Jul 21, 2010
25,770
4,154
elflord said:
neuroanatomist said:
Perhaps one more refinement: irrelevant to camera sensor image quality. A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Well if your point is that the final product depends on aggregate performance of the whole system, that's fine, but what are they to do about it ?

They could aggregate AF performance into an even more subjective and debatable score, but you have repeatedly said that this is precisely what they shouldn't be doing.

Are you simply opposed to them publishing any sensor benchmarks because they expose the technological stagnation of your favourite camera brand ?

Sorry, but my reply wasn't directed generally, nor to you specifically. If you read my previous posts, hopefully you'd realize your questions are tangential and I'm saying none of the statements you seem to be expressing on my behalf.
 

neuroanatomist

I post too Much on Here!!
CR Pro
Jul 21, 2010
25,770
4,154
TheSuede said:
neuroanatomist said:
Perhaps one more refinement: irrelevant to camera sensor image quality. A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Hehe... Yup - true.

But when you can 'assume with reasonable confidence' that two cameras have similar AF accuracy and sensitivity, the static image quality metric still rules.
-And you might have noticed that I deliberately MENTIONED the autofocus as being left out of the metric in the first place - so no modification is needed.

Better image quality = better image quality.
No modifier, no disclaimer, no BS sidetracking.

Well, perhaps a modifier or disclaimer is in order. ;) For example, how about better image quality at ISO 12800? That may be important to some, but not others, and it's not represented in DxOMark's Scores at all. But that's why I've stated several times that I applaud them for making the measurement data available. While I feel it's improper to publish Scores without disclosing full details of how those Scores are determined, personally, it doesn't matter to me for my own buying decisions.
 

elflord

EOS RP
Aug 2, 2011
692
0
neuroanatomist said:
Sorry, but my reply wasn't directed generally, nor to you specifically. If you read my previous posts, hopefully you'd realize your questions are tangential and I'm saying none of the statements you seem to be expressing on my behalf.

After rereading I think I see what you were getting at -- it was just very puzzling on a first reading (and yes I understand that taken literally it was inconsistent with what you've posted before which is why I found it puzzling)
 

bdunbar79

EOS R6
May 16, 2012
3,152
0
42
Wadsworth, OH
Well it's like me and basketball. In HS I couldn't miss in practice from the 3 point arc. So you could say I was a great 3-point shooter. However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open. So it was actually meaningless.

We here on the forum know that this is just a sensor score. Howver, consumers don't know that and if they just read the scores they automagically assume it is a better camera. That's the problem. It isn't. Sensor is just ONE measured aspect of a camera and as a scientist, when I read their crap, it comes off to me as rather unscientific. Whether it is or isn't, it just comes off that way to me.

You can defend it all you want, but it looks like crap to me and I don't use their scores at all. Each to their own.
 

thepancakeman

If at first you don't succeed, don't try skydiving
Aug 18, 2011
476
0
Minnesota
bdunbar79 said:
Well it's like me and basketball. In HS I couldn't miss in practice from the 3 point arc. So you could say I was a great 3-point shooter. However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open. So it was actually meaningless.

You, too, huh? Excellent analogy!
 

tnargs

EOS 90D
Sep 7, 2010
145
3
260 posts, how about a summary? :)

Well, it seems that when people legitimately criticize DxO Labs' application of their measurements, apologists rush in to defend their measurements.

That's called an air ball, guys.

And when people point out that these DxO scores (applications) are being misused by review sites etc in a way that gives the wrong impression, apologists rush in to say that's not really DxO Labs' fault.

You must be joking.

Let us say Mercedes Inc engaged you to independently measure the quality of their product vis a vis say, Ford Inc, which they had also measured and knew would favour them. They also told you to summarize your findings and publish them. You go ahead and do it but goodness gracious, what's this? Your website is technically correct and shows Mercedes' quality advantage at the deepest level, but your way of summarizing and organizing and communicating gives the general impression that Ford is the better overall vehicle!

Suddenly, due to your excellent technical reputation, the world takes notice of your findings. Reviewers start to refer to 'The Ford Advantage' on their websites. Fleet buyers start recommending to boardrooms that current Mercedes contracts for trucks and buses should not be renewed, partly due to a new quality value matrix by an industry technical expert that favours other marques. The boardroom executive gossip soon reaches the ears of Mercedes executives.

Suddenly your client is on the phone to you in an absolute Teutonic fury. Yes, they say, your base measurements are correct but what on earth have you done with it? Why the blazes that way? Don't you realize how everyone is interpreting it?

Here is your answer: (hold on, wait for it, it's a beauty): "It's not my fault if they are not well enough informed to grasp the underlying deep measurements correctly."

pfft

Good luck getting your invoice processed.
 

tron

EOS R5
CR Pro
Nov 8, 2011
4,962
1,316
RLPhoto said:
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
+1000000 Very true. Rumor has it that a camera does not become worse simply because newer models appear ;D ;D ;D
 
P

paul13walnut5

Guest
@bdunbar79
Well it's like me and basketball. In HS I couldn't miss in practice from the 3 point arc. So you could say I was a great 3-point shooter. However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open. So it was actually meaningless.

We here on the forum know that this is just a sensor score. Howver, consumers don't know that and if they just read the scores they automagically assume it is a better camera. That's the problem. It isn't. Sensor is just ONE measured aspect of a camera and as a scientist, when I read their crap, it comes off to me as rather unscientific. Whether it is or isn't, it just comes off that way to me.

You can defend it all you want, but it looks like crap to me and I don't use their scores at all. Each to their own.

Do consumers care? When I bought my first camera I bought a Canon. Why? Because thats what my Dad had used for years, and what my sister used. I think the majority of consumers are the same.

Now I'm a bit more serious about it I read various reviews, get hands on, take in a memory card, try a couple of lenses. I'm not very scientific, and don't photograph charts for my test, or in fact for my work or pleasure.
Maybe I could be more demanding, but I just want a camera that fits my lenses, does decent video and takes decent pics.

My cameras do that for me, I might help out a little with grading or RAW processing now and then, but I'm not losing any sleep what so ever over what Nikon or DxO are doing. I can understand folk at the very top of their game chasing that extra 1% to the nth degree, or whatever. I'm not at the top of my game, and I doubt that most of us here are.

One of my favourite photographers, Maritn Parr, did a lot of his best work on a Nikon F90. He was asked once what kind of lens he used and he had to look at the front of it to see. I mean that as no sleight. I think we just lose track of what is important sometimes. I think this obsession with DxO is psuedo-autistic, and as we are all churning out great images on Canon gear, can deduce it doesn't really mean that much.

I've always thought of DxO as being like reviewing a car that doesn't have a gearbox. I've never seen a difference on a print or on screen that convinces me Canon is as far behind Nikon as the fanboys interpretation of the stats would have you believe.

If I was in the market for a 36MP camera, then I might buy a D800, I'd certainly consider it. But I'm not. To be honest if I could get a DSLR that did video with a stills resolution that permitted straight digital sampling to 1080 (so say a 3840 wide sensor) I would be absolutely delighted. On that basis I will be delighted to see what DxO make of the c100 or c300, as one can only image they'll hate it!
 

Aglet

EOS 5D Mark IV
Feb 26, 2012
1,729
16
AB
RLPhoto said:
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
it was not when it was released
it is not much better after all the firmware updates
it made a big splash being the first CHEAP full-frame and those who wanted or needed that were so enthused with this new toy they paid little regard to its IQ shortcomings because it offered IQ benefits and features previously unavailable.

Endlessly trumpeting on that basis suckered me in to buying one; my most regrettable Canon purchase.
not for lack of due diligence, but for lack of honest and clear information about the 5D2's weaknesses, which became clear enough after I used it for a while. .. and after others began posted about its less than ideal low ISO FPN issues.

If you still find it meets your needs you either lucked out with a good one or you don't mind crushing your blacks a few more levels than some of us.

edit adding DxOmark comment below:

and THIS is what annoys me about DxOmarks results. not just that they assign a vague overall score to a camera, based solely on measured sensor merits, but that they do not adequately disclose the testing criteria and data in a way that would allow the technically astute reader the opportunity to evaluate the data on their own. And, for the most part that I've found, neither do other sensor tech sites.

When FPN affects a sensor to the degree some of the Canon's (and some other mfr's products) were affected, it would have been very valuable to have a good idea whether the noise was random and acceptable or whether it was patterned and possibly objectionable to prospective purchasers.

DxOmark's data is useful but incomplete and that makes it much less useful than it would have been in my particular instance.
 

RLPhoto

Gear doesn't matter, Just a Matter of Convenience.
Mar 27, 2012
3,778
0
San Antonio, TX
www.Ramonlperez.com
Aglet said:
RLPhoto said:
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
it was not when it was released
it is not much better after all the firmware updates
it made a big splash being the first CHEAP full-frame and those who wanted or needed that were so enthused with this new toy they paid little regard to its IQ shortcomings because it offered IQ benefits and features previously unavailable.

Endlessly trumpeting on that basis suckered me in to buying one; my most regrettable Canon purchase.
not for lack of due diligence, but for lack of honest and clear information about its weaknesses, which became clear enough after I used it for a while. .. and after others began posted about its less than ideal low ISO FPN issues.

If you still find it meets your needs you either lucked out with a good one or you don't mind crushing your blacks a few more levels than some of us.

*Facepalm

If your ultra-mega analytical about bringing up black's 4-stops, which by the way is why they are called black's not midtones, then you have some other serious issues to contend with. (IE: Timing for Ideal Light)

I'd have 7D landscapes that are heavily processed and pushed to +3 in the corners than were accepted for Istock. Thats some very strict standards.