April 18, 2014, 11:02:10 AM

Author Topic: DxOMark vs. Reality  (Read 70745 times)

Meh

  • 5D Mark III
  • ******
  • Posts: 679
    • View Profile
Re: DxOMark vs. Reality
« Reply #15 on: October 09, 2012, 03:15:11 PM »
why should i, as photographer, care about how well nikon or canon sells?

even as a shop owner i don´t care about THEIR numbers.. i care about MY numbers.

as a customer i care what the camera offers me for my money... nothing more nothing less.

canon can outsell nikon 10x times.. that will still not make their current lineup better.

Very true.  However, if the reason why Canon outsells Nikon is because they have an overall better offering and better value proposition then your point is fully consistent with the market sentiment.  On the other hand, if Canon is outselling other brands because of simple popularity or a larger marketing budget then perhaps you could conclude they are not better despite the numbers.  Markets can get it wrong, but typically the markets get it right over the long-term.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #15 on: October 09, 2012, 03:15:11 PM »

dtaylor

  • 7D
  • *****
  • Posts: 441
    • View Profile
Re: DxOMark vs. Reality
« Reply #16 on: October 09, 2012, 03:43:37 PM »
I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices  ;)

elflord

  • 5D Mark III
  • ******
  • Posts: 705
    • View Profile
Re: DxOMark vs. Reality
« Reply #17 on: October 09, 2012, 08:04:12 PM »
So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

There's much more to a camera system than sensor technology.

The willingness and ability to stand behind ones product is very important if you're taking the long view. Orphaned products are almost worthless. No-one besides Nikon comes close and when their main advantage over Canon is a product that is sourced from a competitor, you can't give them too much credit for it.

Sensor technology does count for something, Canon need to be respectable but the evidence is that other factors are more important.






lfg530

  • Guest
Re: DxOMark vs. Reality
« Reply #18 on: October 09, 2012, 09:28:55 PM »
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).

It doesn't take long to see where Nikon is lacking. Just consult the ISO charts. There are sites that give you the side by side. No Lecia comparison, just side by side Nikon vs Canon.
Case in point, 6D vs the D600. Comparable offerings right?
Compare the kit lenses that will be offered with both cameras. Nikon gives you a better sensor, Canon does it with better glass. In three years the bodies are old news, obsolete by gear head standards. At that point who is ahead, Canon or Nikon? It seems the sensor race goes back and forth. Good Glass is Good Glass no matter which sensor is better.

Still dodging proofs and bringing empty assumptions. You're talking about ONE example and it's a kit lens, the 24-105 is great and probably (didn't test it myself) better than the 24-85 (wich is no slouch either at half the price....). But I tought we were talking about the lens lineup in general, not specific cases. And it's possible with both brands to have a really nice lens lineup if you invest in glass instead of bodies and both brands will offer great bodies when the times comes to upgrade...

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #19 on: October 09, 2012, 09:43:04 PM »
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).

It doesn't take long to see where Nikon is lacking. Just consult the ISO charts. There are sites that give you the side by side. No Lecia comparison, just side by side Nikon vs Canon.
Case in point, 6D vs the D600. Comparable offerings right?
Compare the kit lenses that will be offered with both cameras. Nikon gives you a better sensor, Canon does it with better glass. In three years the bodies are old news, obsolete by gear head standards. At that point who is ahead, Canon or Nikon? It seems the sensor race goes back and forth. Good Glass is Good Glass no matter which sensor is better.

Still dodging proofs and bringing empty assumptions. You're talking about ONE example and it's a kit lens, the 24-105 is great and probably (didn't test it myself) better than the 24-85 (wich is no slouch either at half the price....). But I tought we were talking about the lens lineup in general, not specific cases. And it's possible with both brands to have a really nice lens lineup if you invest in glass instead of bodies and both brands will offer great bodies when the times comes to upgrade...

See how empty this is.

Lets look at the top side of the line up. This will take it from talking about 1 lens to 10.
Hop over to TDP's ISO charts  and compare any of the big supertele primes 200mm or up, version I or version II.
If you find a Nikon that outperforms any of its Canon's matches old version or new let me know. I haven't compared one yet that did.
A point to consider, the Nikon lenses ISO on the site were shot with the "superior" Nikon sensor.

Sure you can point out, the Canon lenses cost more. Canon knows there long lenses are better, they demand a premium and people pay it.

Which direction should we go next, I am sure there is a Nikon nugget somewhere. But, I know there are good lenses made for Nikon cameras. Just look up Zeiss.

tnargs

  • Rebel T5i
  • ****
  • Posts: 138
    • View Profile
Re: DxOMark vs. Reality
« Reply #20 on: October 09, 2012, 09:56:20 PM »
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.

dtaylor

  • 7D
  • *****
  • Posts: 441
    • View Profile
Re: DxOMark vs. Reality
« Reply #21 on: October 09, 2012, 10:16:35 PM »
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.

I would be curious to see that if you have a link. If true, they need to post this in big, bold type on every page.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #21 on: October 09, 2012, 10:16:35 PM »

neuroanatomist

  • CR GEEK
  • *******
  • Posts: 12774
    • View Profile
Re: DxOMark vs. Reality
« Reply #22 on: October 09, 2012, 11:05:49 PM »
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.


It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page, which is easily accesed from the About tab.  Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance."  Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure.  But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 

BTW, putting it in big bold type on every page wouldn't help.  They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Hillsilly

  • 5D Mark III
  • ******
  • Posts: 678
    • View Profile
Re: DxOMark vs. Reality
« Reply #23 on: October 09, 2012, 11:07:49 PM »
Some valid points above and its been interesting reading the discussions about DxOMark in recent months and gaining a better understanding of their tests.  I used to find them very influential - A year or two ago, I was considering purchasing a little Pentax kit for travelling (DSLR and a couple of pancake lenses).  But DxO gave their lenses such bad ratings that I gave up on the idea.  And I've got almost a basic understanding of things.  I don't know what complete beginners make of it all.  Imagine how good Canon sales would be if they were competitve in DxO rankings.
1000FN | 7E | 3000 | 3 | LS-100TS

PackLight

  • Guest
Re: DxOMark vs. Reality
« Reply #24 on: October 09, 2012, 11:11:12 PM »
BTW, putting it in big bold type on every page wouldn't help.  They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?

At least one, the guy that is always in front of me.

lfg530

  • Guest
Re: DxOMark vs. Reality
« Reply #25 on: October 10, 2012, 01:33:57 AM »
See how empty this is.

Lets look at the top side of the line up. This will take it from talking about 1 lens to 10.
Hop over to TDP's ISO charts  and compare any of the big supertele primes 200mm or up, version I or version II.
If you find a Nikon that outperforms any of its Canon's matches old version or new let me know. I haven't compared one yet that did.
A point to consider, the Nikon lenses ISO on the site were shot with the "superior" Nikon sensor.

Sure you can point out, the Canon lenses cost more. Canon knows there long lenses are better, they demand a premium and people pay it.

Which direction should we go next, I am sure there is a Nikon nugget somewhere. But, I know there are good lenses made for Nikon cameras. Just look up Zeiss.
[/quote]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.

Otara

  • PowerShot G16
  • **
  • Posts: 35
    • View Profile
Re: DxOMark vs. Reality
« Reply #26 on: October 10, 2012, 04:01:21 AM »
Noone thinks it might just be that Canon is mostly selling 18MP sensors, and Nikon is mostly selling 16MP sensors at the low end of the DSLR market?

The market tends to go for simple differentiation rather than the more complex arguments Im seeing, particularly when there isnt an easier way to tell which one is really better.

Otara

davidpeter

  • Guest
Re: DxOMark vs. Reality
« Reply #27 on: October 10, 2012, 04:12:56 AM »
DxO, here we go again.

I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

For the reasonable ones, they have well detailed protocols, and measurement methods:

http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements
http://www.dxo.com/us/embedded_imaging/scientific_publications

And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

But you have to understand, that this is a pixel based data (so when two sensors share the same SNR, the one with the bigger MP will give better IQ downsized to a given resolution). They try to compensate this in the scores, but score vs. charts is like photoshopped jpeg vs. RAW. The first may look better, but the second carries the more information.

I always smile, when see comments "DR tested by me". Yep, and how? Because that is the question. At DxO, we know that exactly.

But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.


Finally, let me have some observation on a comment, which made me writing this one:

Quote
* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs are expensive, as they are:
Big in size. In Si technology, price grows exponentially with size. That's why they even glue the sensor form many pieces some times.
They are rare. Mass production would bring prices down to it's quarter.

Quote
* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests. And we already talked about the question of resolution.

So, i tried once more, I promise this was my last attempt on this forum, unless asked...

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #27 on: October 10, 2012, 04:12:56 AM »

weixing

  • Rebel T5i
  • ****
  • Posts: 189
    • View Profile
Re: DxOMark vs. Reality
« Reply #28 on: October 10, 2012, 04:40:03 AM »
Hi,
   IMHO, DxO comparison will be valid if RAW file is RAW data, but in reality RAW file is not RAW data especially Nikon RAW file. IMHO, DxO should be ok to compare camera models from same brand since most likely the way they handle the RAW data will be similar.

   Have a nice day.
 
« Last Edit: October 10, 2012, 04:42:18 AM by weixing »

dtaylor

  • 7D
  • *****
  • Posts: 441
    • View Profile
Re: DxOMark vs. Reality
« Reply #29 on: October 10, 2012, 04:47:39 AM »
DxO, here we go again.

DxO defender, here we go again.

Quote
If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly.

Ergo DxO is silly because that is exactly what they claim can be done when they publish their scores. I will stop hammering them for their stupid overall scores when they remove them all from the site.

Quote
And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

I contest this point. At least as far as dynamic range is concerned, there are repeat instances where they are clearly wrong.

Quote
But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.

Are you part of DxO? Well...we will stop "wasting your time" when you stop wasting ours publishing silly overall scores that are repeated ad nauseam in forums. And when you correct some of the obvious flaws in your test results.
Quote
Quote
* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.
Medium format backs are expensive...

Yes, they are. Their IQ also wipes the floor with consumer APS-C equipment. (And that's coming from a huge fan of today's APS-C sensors!) And DxO looks stupid for claiming otherwise.

Quote
Quote
* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).
Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Quote
Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)
The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests.

If you work for DxO, do the entire team a favor: buy a transmission step wedge and use it. Don't run it through a flawed computer analysis. Actually use it and eyeball the output. You will be embarrassed at some of the mistakes in your database.

canon rumors FORUM

Re: DxOMark vs. Reality
« Reply #29 on: October 10, 2012, 04:47:39 AM »