canon rumors FORUM

Gear Talk => Canon General => Topic started by: neuroanatomist on October 09, 2012, 09:13:17 AM

Title: DxOMark vs. Reality
Post by: neuroanatomist on October 09, 2012, 09:13:17 AM
There are lots of posts about DxOMark, lots of support, lots of bashing, some of it fact-based, much of it hearsay and knee-jerk reaction.  There are many posts about sales figures, generally fewer than about DxOMark, and generally with even less of them with real facts.

So...let's do a little reality check, about the past 5 years in DxOMark-land and the past 5 years in the real world.

The year is 2007.  Nikon has the D3 as a flagship and the D300 as the prosumer APS-C body.  Canon has the 1DsIII and D40, respectively.  DxOMark scores the D3 as 81, the 1DsIII as 80 - a meaningless difference, by their standards.  The D300 scores a 67, while the 40D scores a 64 - that is starting to look like a real difference. 

2008 saw Nikon enter the 'affordable' FF market with the D700, which DxOMark scores an insignificant 1 point higher than Canon's 5DII at 80 vs. 79.  2009 had updates to the high-end prosumer APS-C segment, and Nikon's D300S scores a 70, while Canon's 7D scores a 66.  2010 brought us mid-range APS-C updates, and while the 60D with the 7D's sensor scores a 66, the D7000 scores an 80, beating Canon's then-current FF sensors.

This year, the somewhat-less-affordable FF battle continues, with the 5DIII scoring 81, barely better than it's predecessor, while the D800/E scores a very impressive 95/96, soundly beating all Canon cameras as well as every other sensor DxOMark has ever tested, including medium format bodies with sensors with 2.5-times larger sensors.

So, it seems quite clear that over the past 5 years in DxOMark-land, Nikon has been dramatically improving, Canon has been stagnating, and Nikon is way ahead.  But that's DxOMark-land.  What about the real world?

Canon and Nikon are publicly traded companies, and as such, they publish their annual and quarterly reports (in the Investor Relations sections of their corporate websites).  IDC (a market analysis firm) also publishes annual summaries of the market as a whole.

In 2007 (http://www.imaging-resource.com/NEWS/1207604859.html), Canon had 43% of the dSLR market, Nikon had 40%.  In 2010 (http://www.bloomberg.com/news/2011-04-15/sony-nikon-narrow-gap-to-canon-with-new-digital-camera-models.html), Canon had 44.5% of the dSLR market, Nikon had 29.8%.  So, over years while Nikon was bringing us better sensors than Canon, Nikon was losing market share while Canon was gaining it.  Ok, fine, but that's 2010.  What about this year?

In 2Q2012 (by calendar year, they report it as 1QFY2013 (http://www.nikon.com/about/ir/ir_library/result/pdf/2013/13first_1_e.pdf)), Nikon had an 18% y/y growth of unit sales of dSLRs and lenses.  In 2Q2012 (http://www.canon.com/ir/conf2012q2/conf2012q2e.pdf), (they report by CY, not FY), Canon had a 47% y/y growth of unit sales of dSLRs and lenses.

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.
Title: Re: DxOMark vs. Reality
Post by: Mt Spokane Photography on October 09, 2012, 12:59:19 PM
Part of this is because lenses play a very important role in photography, and Nikon has lagged behind with FF lenses.  They have very few really good lenses, their 14-24 being the best example followed by their new 85mm f/1.4.  Their 24-70mm has horible CA, their 70-200mm is very good.  There are a ton of "D" lenses that are good, but no one seems to want them, and they do not have coatings that are well suited to digital.

Then, Nikon nothing that matches Canons 24-105mm L, 135mmL, or 100-400mmL much less the Canon 400mm f/5.6. 

Nikon has been cranking out new low cost DX lenses as though the lens side does not believe FF is the way to go. I get the feeliing that they do not have a coordinated plan.  If they are bring excellent new FF bodies to the market while developing DX lenses, customers like me who went out and bought a D800 and 24-70 f/2.8, 80-200mm f/2.8, 200-400mm f/4 and discovered that I'd have to buy a Sigma lens if I wanted a decent low cost 400mm lens that could not match the older Canon designs, and found nothing to match my favorite Canon lenses.  I finally sold the body and the lenses, and bought another new Canon body.
The Nikon lens prices are generally too high for what you get.
Its a shame, but Nikon does seem to be struggling, and I'd credit their being behind in good lens availability at low prices for a big part of it.

With Sigma agressively churning out ever new and improved lens designs, I'd think this helps Nikon but buyers generally prefer to stick with OEM lenses.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 01:11:13 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.
Title: Re: DxOMark vs. Reality
Post by: bornshooter on October 09, 2012, 01:20:00 PM
DXO mark means nothing to me... i will say glass is what matters and canon's ef range of lenses are untouchable they just make my mouth water i want them all but i will need to sell my girlfriend lol
Title: Re: DxOMark vs. Reality
Post by: Mt Spokane Photography on October 09, 2012, 01:28:35 PM
DXO mark means nothing to me... i will say glass is what matters and canon's ef range of lenses are untouchable they just make my mouth water i want them all but i will need to sell my girlfriend lol
Could you post a photo (of her) :)
Title: Re: DxOMark vs. Reality
Post by: well_dunno on October 09, 2012, 01:34:40 PM
Regarding the glass, DxO is probably the only source that suggests 70-200 f/2.8 IS mkII is less sharp than mark I (suggesting mkII is better in the center but overall mkI is better, if my memory is not failing me). I do not think that affected the 70-200 f/2.8 mkII sales either, rather the effect was on the credibility of the DxO...

Cheers!
Title: Re: DxOMark vs. Reality
Post by: willis on October 09, 2012, 01:40:49 PM
Regarding the glass, DxO is probably the only source that suggests 70-200 f/2.8 IS mkII is less sharp than mark I (suggesting mkII is better in the center but overall mkI is better, if my memory is not failing me). I do not think that affected the 70-200 f/2.8 mkII sales either, rather the effect was on the credibility of the DxO...

Cheers!
And I've always heard that MK2 is superb sharp even sharper than MK1.
Title: Re: DxOMark vs. Reality
Post by: lfg530 on October 09, 2012, 01:57:43 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 02:10:52 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.
Title: Re: DxOMark vs. Reality
Post by: Gothmoth on October 09, 2012, 02:12:15 PM
why should i, as photographer, care about how well nikon or canon sells?

even as a shop owner i don´t care about THEIR numbers.. i care about MY numbers.

as a customer i care what the camera offers me for my money... nothing more nothing less.

canon can outsell nikon 10x times.. that will still not make their current lineup better.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 02:17:09 PM
why should i, as photographer, care about how well nikon or canon sells?

even as a shop owner i don´t care about THEIR numbers.. i care about MY numbers.

as a customer i care what the camera offers me for my money... nothing more nothing less.

canon can outsell nikon 10x times.. that will still not make their current lineup better.

The shop owner point is an interesting one.

The local shop pushes Nikon. I asked why, what the owner told me is that Canon controls what he can sell his lenses and cameras for. He gets a bigger margin out of selling Nikon's.

After the purchase they push the off brand lenses even harder because the margin is greater. There is one color that Nikon processes better than Canon, and keeps them in the game. It is green.
Title: Re: DxOMark vs. Reality
Post by: Chewy734 on October 09, 2012, 02:45:40 PM
That is interesting...  my local AD has lots of banners and large signs outside their storefront...  all are Nikon, none are Canon.  Inside the two biggest display shelves are Nikon and Canon.  Every time I have gone in (probably a couple times a week, to once a month), I've never seen anyone actually buy anything Nikon.  But, I've seen a lot of people buy Canon gear.  I'm not saying that they aren't selling a lot of Nikon gear, because I'm sure they are, but those are just my observations.

I once asked the rental section of the store a similar question...  what do you rent out more often?  The answer was Canon equipment 3x as often.  Actually, Nikon was only rented slightly more often than Mamiya, Leaf, and the Hassys.  In fact Zeiss lenses were more often rented than Nikon lenses.

What's interesting is that when I went to the Browns game last month, there were only two photographers shooting Nikon.  The other dozen were shooting Canon.  Why?

But then, last year I went to Belize for vacation.  While walking around, going on excursions, etc, I did not see one other Canon...  not one...  not even a tiny PowerShot.  All I saw were Nikon dSLRs, compacts, etc.  I was the only one I saw shooting with my Canon (5D2 at the time).

So, what's this all mean?  Who knows.
Title: Re: DxOMark vs. Reality
Post by: lfg530 on October 09, 2012, 02:46:46 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).
Title: Re: DxOMark vs. Reality
Post by: bvukich on October 09, 2012, 03:04:58 PM
But then, last year I went to Belize for vacation.  While walking around, going on excursions, etc, I did not see one other Canon...  not one...  not even a tiny PowerShot.  All I saw were Nikon dSLRs, compacts, etc.  I was the only one I saw shooting with my Canon (5D2 at the time).

So, what's this all mean?  Who knows.

It's Ashton Kutcher... he's so dreamy...

 :P
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 03:10:05 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).

It doesn't take long to see where Nikon is lacking. Just consult the ISO charts. There are sites that give you the side by side. No Lecia comparison, just side by side Nikon vs Canon.
Case in point, 6D vs the D600. Comparable offerings right?
Compare the kit lenses that will be offered with both cameras. Nikon gives you a better sensor, Canon does it with better glass. In three years the bodies are old news, obsolete by gear head standards. At that point who is ahead, Canon or Nikon? It seems the sensor race goes back and forth. Good Glass is Good Glass no matter which sensor is better.
Title: Re: DxOMark vs. Reality
Post by: Meh on October 09, 2012, 03:15:11 PM
why should i, as photographer, care about how well nikon or canon sells?

even as a shop owner i don´t care about THEIR numbers.. i care about MY numbers.

as a customer i care what the camera offers me for my money... nothing more nothing less.

canon can outsell nikon 10x times.. that will still not make their current lineup better.

Very true.  However, if the reason why Canon outsells Nikon is because they have an overall better offering and better value proposition then your point is fully consistent with the market sentiment.  On the other hand, if Canon is outselling other brands because of simple popularity or a larger marketing budget then perhaps you could conclude they are not better despite the numbers.  Markets can get it wrong, but typically the markets get it right over the long-term.
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 09, 2012, 03:43:37 PM
I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices  ;)
Title: Re: DxOMark vs. Reality
Post by: elflord on October 09, 2012, 08:04:12 PM
So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

There's much more to a camera system than sensor technology.

The willingness and ability to stand behind ones product is very important if you're taking the long view. Orphaned products are almost worthless. No-one besides Nikon comes close and when their main advantage over Canon is a product that is sourced from a competitor, you can't give them too much credit for it.

Sensor technology does count for something, Canon need to be respectable but the evidence is that other factors are more important.





Title: Re: DxOMark vs. Reality
Post by: lfg530 on October 09, 2012, 09:28:55 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).

It doesn't take long to see where Nikon is lacking. Just consult the ISO charts. There are sites that give you the side by side. No Lecia comparison, just side by side Nikon vs Canon.
Case in point, 6D vs the D600. Comparable offerings right?
Compare the kit lenses that will be offered with both cameras. Nikon gives you a better sensor, Canon does it with better glass. In three years the bodies are old news, obsolete by gear head standards. At that point who is ahead, Canon or Nikon? It seems the sensor race goes back and forth. Good Glass is Good Glass no matter which sensor is better.

Still dodging proofs and bringing empty assumptions. You're talking about ONE example and it's a kit lens, the 24-105 is great and probably (didn't test it myself) better than the 24-85 (wich is no slouch either at half the price....). But I tought we were talking about the lens lineup in general, not specific cases. And it's possible with both brands to have a really nice lens lineup if you invest in glass instead of bodies and both brands will offer great bodies when the times comes to upgrade...
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 09:43:04 PM
Neuro your post could have been sumed up in just a few words.

"Canon Glass Rules"

We can debate the sensors of the day and in a few months the conversation will change, but in 8 years we will still be debating the same glass. Maybe in those 8 years Nikon will come up with some glass worthy of a debate.


Seriously? Their 24-70 is really great, way better than the mk1 was. Their 1.4 primes are about as good as the canons are (the canon 24 is excellent tough). Both 70-200 are great. Nikon recently came out with 3 new 1.8g lenses that are awesome and affordable... We're still not talking about the 14-24 or the ridiculous amount of old F mount lenses (e.g the 135 DC or the 55 micro) that are still awesome pieces of glass. Yes Canon also has great glass and some lenses that don't have good nikon counteparts, but nikon is not out of the game in that regard.

I'm really having a problem seeing where the nikon glass is unworthy outside of a fanboy perspective.

Thats funny, lets go back to talking about sensors. That way the Nikonians can particapate in the conversation.

Nice to see you're open minded and bringing proof to the conversation; you don't seem to be a mindless Canonian at all.

Just sayin': I'm the proud owner of a 7d and macro 100L wich I loove, I tried several canon primes and L zooms and most of them were great, but I also compared to equivalents in nikon and their glass has nothing to be ashamed of. It's really cute to see some of you guys acting like canon glass is like an affordable leica kind of thing, but it's not and nikon is serious competition in that matter (at least for what I tested MYSELF and any review site you'll find that compare both).

It doesn't take long to see where Nikon is lacking. Just consult the ISO charts. There are sites that give you the side by side. No Lecia comparison, just side by side Nikon vs Canon.
Case in point, 6D vs the D600. Comparable offerings right?
Compare the kit lenses that will be offered with both cameras. Nikon gives you a better sensor, Canon does it with better glass. In three years the bodies are old news, obsolete by gear head standards. At that point who is ahead, Canon or Nikon? It seems the sensor race goes back and forth. Good Glass is Good Glass no matter which sensor is better.

Still dodging proofs and bringing empty assumptions. You're talking about ONE example and it's a kit lens, the 24-105 is great and probably (didn't test it myself) better than the 24-85 (wich is no slouch either at half the price....). But I tought we were talking about the lens lineup in general, not specific cases. And it's possible with both brands to have a really nice lens lineup if you invest in glass instead of bodies and both brands will offer great bodies when the times comes to upgrade...

See how empty this is.

Lets look at the top side of the line up. This will take it from talking about 1 lens to 10.
Hop over to TDP's ISO charts  and compare any of the big supertele primes 200mm or up, version I or version II.
If you find a Nikon that outperforms any of its Canon's matches old version or new let me know. I haven't compared one yet that did.
A point to consider, the Nikon lenses ISO on the site were shot with the "superior" Nikon sensor.

Sure you can point out, the Canon lenses cost more. Canon knows there long lenses are better, they demand a premium and people pay it.

Which direction should we go next, I am sure there is a Nikon nugget somewhere. But, I know there are good lenses made for Nikon cameras. Just look up Zeiss.
Title: Re: DxOMark vs. Reality
Post by: tnargs on October 09, 2012, 09:56:20 PM
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 09, 2012, 10:16:35 PM
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

Post #1 doesn't seem to recognise this.

I would be curious to see that if you have a link. If true, they need to post this in big, bold type on every page.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 09, 2012, 11:05:49 PM
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page (http://www.dxomark.com/index.php/About/Sensor-scores), which is easily accesed from the About tab.  Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance."  Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure.  But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 

BTW, putting it in big bold type on every page wouldn't help.  They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?
Title: Re: DxOMark vs. Reality
Post by: Hillsilly on October 09, 2012, 11:07:49 PM
Some valid points above and its been interesting reading the discussions about DxOMark in recent months and gaining a better understanding of their tests.  I used to find them very influential - A year or two ago, I was considering purchasing a little Pentax kit for travelling (DSLR and a couple of pancake lenses).  But DxO gave their lenses such bad ratings that I gave up on the idea.  And I've got almost a basic understanding of things.  I don't know what complete beginners make of it all.  Imagine how good Canon sales would be if they were competitve in DxO rankings.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 09, 2012, 11:11:12 PM
BTW, putting it in big bold type on every page wouldn't help.  They put big, black Speed Limit signs on all the roads, ho many people actually drive the speed limit or below?

At least one, the guy that is always in front of me.
Title: Re: DxOMark vs. Reality
Post by: lfg530 on October 10, 2012, 01:33:57 AM
See how empty this is.

Lets look at the top side of the line up. This will take it from talking about 1 lens to 10.
Hop over to TDP's ISO charts  and compare any of the big supertele primes 200mm or up, version I or version II.
If you find a Nikon that outperforms any of its Canon's matches old version or new let me know. I haven't compared one yet that did.
A point to consider, the Nikon lenses ISO on the site were shot with the "superior" Nikon sensor.

Sure you can point out, the Canon lenses cost more. Canon knows there long lenses are better, they demand a premium and people pay it.

Which direction should we go next, I am sure there is a Nikon nugget somewhere. But, I know there are good lenses made for Nikon cameras. Just look up Zeiss.
[/quote]

Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.
Title: Re: DxOMark vs. Reality
Post by: Otara on October 10, 2012, 04:01:21 AM
Noone thinks it might just be that Canon is mostly selling 18MP sensors, and Nikon is mostly selling 16MP sensors at the low end of the DSLR market?

The market tends to go for simple differentiation rather than the more complex arguments Im seeing, particularly when there isnt an easier way to tell which one is really better.

Otara
Title: Re: DxOMark vs. Reality
Post by: davidpeter on October 10, 2012, 04:12:56 AM
DxO, here we go again.

I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

For the reasonable ones, they have well detailed protocols, and measurement methods:

http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements (http://www.dxomark.com/index.php/About/In-depth-measurements/Measurements)
http://www.dxo.com/us/embedded_imaging/scientific_publications (http://www.dxo.com/us/embedded_imaging/scientific_publications)

And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

But you have to understand, that this is a pixel based data (so when two sensors share the same SNR, the one with the bigger MP will give better IQ downsized to a given resolution). They try to compensate this in the scores, but score vs. charts is like photoshopped jpeg vs. RAW. The first may look better, but the second carries the more information.

I always smile, when see comments "DR tested by me". Yep, and how? Because that is the question. At DxO, we know that exactly.

But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.


Finally, let me have some observation on a comment, which made me writing this one:

Quote
* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.
Medium format backs are expensive, as they are:
Big in size. In Si technology, price grows exponentially with size. That's why they even glue the sensor form many pieces some times.
They are rare. Mass production would bring prices down to it's quarter.

Quote
* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).
Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)
The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests. And we already talked about the question of resolution.

So, i tried once more, I promise this was my last attempt on this forum, unless asked...
Title: Re: DxOMark vs. Reality
Post by: weixing on October 10, 2012, 04:40:03 AM
Hi,
   IMHO, DxO comparison will be valid if RAW file is RAW data, but in reality RAW file is not RAW data especially Nikon RAW file. IMHO, DxO should be ok to compare camera models from same brand since most likely the way they handle the RAW data will be similar.

   Have a nice day.
 
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 10, 2012, 04:47:39 AM
DxO, here we go again.

DxO defender, here we go again.

Quote
If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly.

Ergo DxO is silly because that is exactly what they claim can be done when they publish their scores. I will stop hammering them for their stupid overall scores when they remove them all from the site.

Quote
And they have the charts. This is the data. If you know, what to look for, DxO is a valuable source of information, which can not be compared to any other review, as this is standardized and reproducible.

I contest this point. At least as far as dynamic range is concerned, there are repeat instances where they are clearly wrong.

Quote
But again: if you don't go for the charts, you waste your time. And ours as well, as we have to read all this "biased", "crap", "not reflecting reality" comments, which are based on the lack of knowledge.

Are you part of DxO? Well...we will stop "wasting your time" when you stop wasting ours publishing silly overall scores that are repeated ad nauseam in forums. And when you correct some of the obvious flaws in your test results.
Quote
Quote
* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.
Medium format backs are expensive...

Yes, they are. Their IQ also wipes the floor with consumer APS-C equipment. (And that's coming from a huge fan of today's APS-C sensors!) And DxO looks stupid for claiming otherwise.

Quote
Quote
* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).
Nonlinearity? Yes, it ruins the uniform sensitivity, but it exists, whatever you do.

Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Quote
Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)
The "about the same" is 0,7 EV difference, not to mention, that I would be interested in those tests.

If you work for DxO, do the entire team a favor: buy a transmission step wedge and use it. Don't run it through a flawed computer analysis. Actually use it and eyeball the output. You will be embarrassed at some of the mistakes in your database.
Title: Re: DxOMark vs. Reality
Post by: Hillsilly on October 10, 2012, 05:04:02 AM
DxO Analyser is not made for the photographers. It is for the engineers and technicians.

If you think, that a sensor (and I said sensor, not camera) can be fully described with an overall score in 0-100, you are silly. That is only an advertisement stuff for people, who are not educated in this topic.

And I think that's where the problem lies.  Very few people care or want to know about the technical side.  They just want to take great photos.  To do this you need a camera and if you're doing some research you will come across the DxO site.  How awesome is this!!  Not only do they rank all of the cameras but you can also do a comparison between them.  Despite disclaimers and fine print, you really get the impression that the DxO Mark is an objective assessment of one cameras overall ability vs another.

Anyways, I'm only anti-DxO because my camera gets a lowly "73".  Whereas my Nikon D5100 wielding sister gets an "80".  Yet my camera is sooooo much better....


Title: Re: DxOMark vs. Reality
Post by: aj1575 on October 10, 2012, 06:09:32 AM
Actually, there is only one mistake that DxOMark makes. They should drop the sensor ranking / rating. They do a great job with all the measurements and analysis, and the graphs are very informative. But the rating does not make sense; even the new one, where they try to differentiate between portrait, landscape and high ISO does not really make sense.
Just leave the graphs there, and people can find their own conclusion.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 06:16:29 AM
Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

Medium format backs do not perform well at high ISOs. A "general" rating does not work with a special purpose tool.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera. 

I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

Quote
* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.


Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same". The 7D is substantially better than the 10D for example (9 points in the overall score,  dynamic range substantiially better)

Quote
For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?
Title: Re: DxOMark vs. Reality
Post by: Canon-F1 on October 10, 2012, 06:50:59 AM
What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.    ;D



Title: Re: DxOMark vs. Reality
Post by: straub on October 10, 2012, 07:46:53 AM
This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.
Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.
Title: Re: DxOMark vs. Reality
Post by: Maui5150 on October 10, 2012, 07:47:26 AM
I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

* Report physically unachievable values for dynamic range (i.e. >14 stops from a 14-bit ADC).

* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

All that said...I wish Canon would lower their prices  ;)

This +1000

Most people, especially Professionals in a field, are not fools.

What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

In fact, if DxOMarks is to be trusted, then their should be screams that PhaseOne are TOTAL PIECES OF CRAP and that NIKON is ripping off people by selling any camera above $3000, because the D800 is the ONLY CAMERA anyone should buy willing to spend more than $3000 and those under should go for the D600. 

Not saying they are not good cameras, but what the last year plus has highlighted to me, DxOMarks Scores are COMPLETELY IRRELEVANT AND MEANINGLESS.

An erroneous equation provides erroneous results
Title: Re: DxOMark vs. Reality
Post by: straub on October 10, 2012, 07:48:17 AM
what are DXO´s some would ask.    ;D

Well, they have the word "Science" in their logo and Reichmann doesn't :-)
Title: Re: DxOMark vs. Reality
Post by: Maui5150 on October 10, 2012, 08:00:22 AM

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 


Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations.  So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are.  Anyone can produce DATA.  But data is not information. 

What DxOMarks lacks is RELEVANCE

Title: Re: DxOMark vs. Reality
Post by: M.ST on October 10, 2012, 08:31:19 AM
DxOMarks are useless and only made to entertain hobby photographers.
Title: Re: DxOMark vs. Reality
Post by: davidpeter on October 10, 2012, 09:02:02 AM
Ok, I take this as asking:

Quote
Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do. Heard about DNL and INL, offset and gain errors? It would be a great thing to get rid of this mess, but they keep coming up.

(And yes, you can not gain DR by oversampling, but you can improve the SNR and the number of effective bits, this is how all sigma-delta converters are working)

This is my other favorite:

Quote
What DxOMarks essentially says is ANYONE who has bought a D3x or D4 is a FOOL and wasted their money. when they could have gotten a D800 or D600 for THOUSANDS LESS.

Where do you come from? Don't you have education there? I'm really interested, how can people end up with such an incredibly stupid conclusion...
Title: Re: DxOMark vs. Reality
Post by: tnargs on October 10, 2012, 09:19:30 AM
IIRC, the DxOMark website says (somewhere) that their sensor scores can only be used to compare sensors of the same resolution (MP). So, first decide the resolution of sensor you are interested in (need), then compare sensors of that resolution.

It's a great point, and thanks for mentioning it!

@dtaylor, it's at the bottom of this linked page (http://www.dxomark.com/index.php/About/Sensor-scores), which is easily accesed from the About tab.  Of course, that page also states, "Sensor Overall Score AND resolution are two independent metrics of sensor performance."  Since the Sensor Overall Score is based on image data normalized to a fixed resolution (8 MP), and since the greater the resolution of the sensor relative to that fixed value, the greater the differential impact of that normalization, that would seem to make the Sensor Overall Score a dependent measure, not an independent measure.  But it's been a while since I took basic statistics, so maybe they've changed the definition of an independent measure since then...

well spotted! Half way down the same page says "So before comparing cameras with Sensor Overall Score, it is important to first determine the resolution you are looking for (which largely depends on the size of the screen or the print you intend to use or produce). Once you choose an appropriate resolution, the Sensor Overall Score becomes a fair and powerful tool with which to make comparisons." My memory was pretty good from a couple of years back!

Quote
@tnargs, if anything, that just makes the main point of post #1 even stronger. 

yes, my point
Title: Re: DxOMark vs. Reality
Post by: tnargs on October 10, 2012, 09:22:44 AM
....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 10, 2012, 09:48:30 AM
What has Canon sales to do with DXO ?

Nothing.  Which is exactly my conclusion:

So, DxOMark has said Nikon has had better sensors for years, and the sales data show that Canon has sold more dSLRs and lenses for those same years, and continues to do so, as of the most recent data available.  The straightforward conclusion from the above is that while DxOMark's Scores have a huge impact on the number of inflammatory posts on Internet discussion boards, they have no meaningful impact on the real world aggregate buying decisions of consumers.

If Consumer Reports gives one product a much higher rating that another product, that usually has a tangible impact on sales, i.e. those reports impact buying decisions.  DxOMark has given Nikon higher ratings than Canon for years, and there does not appear to have been any impact of that on sales, i.e. they have no impact on buying decisions. 

Given that:
a) there are flaws and ambiguity in their scores, not to mention some apparently aberrant results (e.g. 70-200 II),
b) their scores apparently (according to them) should not be used to compare cameras of different resolutions,
c) they are scoring only sensor performance, which is just one part of camera performance,
and d) the reduction of a complex imaging system to a single number is essentially meaningless anyway,

I would argue that not only do DxO's Scores have no impact on buying decisions in aggregte, they should have only a minor impact, if any at all, on personal buying decisions. 

As I've pointed out before, their measurements are usually quite good (I say usually because of the above-referenced 70-200 II issue, where DxOMark are the only ones in the world who seem to think the MkI is better than the MkII, and chose not to test another copy but rather to defend their results, which really doesn't help their credibility).  But their Scores are neither meaningful nor useful.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 09:58:37 AM
Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.


Interesting you said 200mm, before I made that post it was the first lens I did look at. At f/2 the Nikon version is very bad in the corners. Supertele's are designed to be used wide open, wide open the Canon wins.

Since Canon doesn't have a 200-400mm yet, Nikon wins by default.

I compared the top Nikon and Canon 70-200mm before I made the last post, because I knew it is where you would want to go next. Again, in the side by side the Canon wins noticeably. 

The 24mm f/1.4 versions of Canon and Nikon, Nikon is near a match but in no way wins. Even wider the 14mm f/.8 the Nikon lens is much softer. I would give Nikon the edge for wide angle, not because of its lenses but its sensor.

If you are comparing top of the line lenses to top of the line lenses Nikon doesn't win. In some categories it gets close to matching but not wining.

Title: Re: DxOMark vs. Reality
Post by: straub on October 10, 2012, 10:39:34 AM
I took this 5 min ago to show the difference between one of  my 5dmk2 and d800

Congratulations, you are the first one to notice that D800 has more DR than 5D2 at base ISO. First prize is a set of tickets for Titanic on it's maiden voyage across the Atlantic.
Title: Re: DxOMark vs. Reality
Post by: Maui5150 on October 10, 2012, 10:45:14 AM
I took this 5 min ago to show the difference between one of  my 5dmk2 and d800

Ummmm  Why did you not compare the d800 to a 5DMKIII at least.

Maybe I should test my 5DMKII versus my dad's D200 as a similar test.  Any bets?

Title: Re: DxOMark vs. Reality
Post by: aj1575 on October 10, 2012, 10:54:29 AM
I like that all of you make a little test. Go to dpreview and choose the review of a Canon Camera; a EOS650D for example (which has a very low DxO-Rating).
Towards the end of the review there is the comparison tool, you can now compare the EOS650D to every other camera tested on dpreview. Take for example a Sony SLT77, a Nikon D7000 and the Oly OM-D E M5. Now just play around with the tool a little bit, and deceide which one of the camera produces the best pictures. Then go to DxOMark and compare their scores there.
I found that interesting.

To my eyes the best sensor was the Nikon, but just a little bit ahead of the Oly, then comes the Canon with a little distance (even though I liked some aspects of the IQ better than from the Nikon, more details for example), and last is the Sony.

What is your impression? And even more interesting, how do they compare to the DxO-rating?
Title: Re: DxOMark vs. Reality
Post by: Ellen Schmidtee on October 10, 2012, 11:17:40 AM
Part of this is because lenses play a very important role in photography, and Nikon has lagged behind with FF lenses.  They have very few really good lenses, their 14-24 being the best example followed by their new 85mm f/1.4.  Their 24-70mm has horible CA, their 70-200mm is very good.  There are a ton of "D" lenses that are good, but no one seems to want them, and they do not have coatings that are well suited to digital.

I'll bet that ignorance of DxO's (or any other specific site that does that kind of testing) existence plays an even larger part.

My two cousins bought Rebels, and I'll bet dollars to monopoly money that attempting this kind of discussion with them will make them give you the 'I'd rather not know what that is, and will you weirdow please go away?' look.
Title: Re: DxOMark vs. Reality
Post by: jthomson on October 10, 2012, 11:50:56 AM
....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

Confuse photographers and themselves.   ;D
Title: Re: DxOMark vs. Reality
Post by: unfocused on October 10, 2012, 12:10:04 PM
....I know, that most of you simply do not care about facts, but for the rest, who are more open minded, I explain (once more).

DxO Analyser is not made for the photographers. It is for the engineers and technicians. ...

And what do they do with it?

Confuse photographers and themselves.   ;D

Best. Answer. Ever
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 12:16:34 PM
Oh I'm sorry, you just needed to clarify that this was about large teles (take a look at the results for the nikkor new 200-400 or new 200 f2 before talking out of your ass). That I didn't test and I don't need, but I'm happy to admit canon has the better big teles altought they don't "blow out of the water" nikkor teles like you're trying to say. I know the reputation of canon is really high regarding lenses over 4000$ and maybe it's where it counts for you, but not for me...

Not sure if trolling at this point, but go take a look at your precious charts for the 1.4 primes, the 24-70s, the 70-200s the new 1.8 primes, the 14mms; the lenses that matter to a lot of photographers too... The only thing I'm trying to do here is just stopping bullshit that come out of places like these where people just try to comfort thereself with their gear by saying other brands are crap; nikon has several lenses that are as good or better than Canons and vice-versa, deal with it.


Interesting you said 200mm, before I made that post it was the first lens I did look at. At f/2 the Nikon version is very bad in the corners. Supertele's are designed to be used wide open, wide open the Canon wins.

Since Canon doesn't have a 200-400mm yet, Nikon wins by default.

I compared the top Nikon and Canon 70-200mm before I made the last post, because I knew it is where you would want to go next. Again, in the side by side the Canon wins noticeably. 

The 24mm f/1.4 versions of Canon and Nikon, Nikon is near a match but in no way wins. Even wider the 14mm f/.8 the Nikon lens is much softer. I would give Nikon the edge for wide angle, not because of its lenses but its sensor.

If you are comparing top of the line lenses to top of the line lenses Nikon doesn't win. In some categories it gets close to matching but not wining.

I have tested lenses since 1978 and I have access to all Nikon and Canon lenses
Regarding 70-200/2,8, IS VR .  Canons 70-200/2,8ismk2  lens is better than Nikons latest and it is seen mostly out in  the corners .
200/2  they are close as they can be
200-400/4 Nikon excellent in a range of 50-100m and optimized for that range compare to infinity
400/2,8 Nikon sharper than Canons old 400, I have not tested the new one
600/4 excellent
Nikon 14-24 excellent but sensitive   to flare, better at 14mm than  Canons 14mm/2,8MK2 regarding resolution
35/1,4 Nikon overall better,slightly sombrero effect and better towards the edges and corners.
85/1,2 ,  Nikon 85/1,4  equal good, 85/1,2 lot of loCA,  85/1,8 canon excellent for the price , also nikon 85/1,8
Nikons 24-70 better than Canons 24-70 . Canon is  is difficult to evaluate since it is a big difference amoung the copies, MK2 not tested

I think the new versions would be the standard to test against, since it is the current offering. That would apply to the 24-70 and the 400mm f/2.8.

I can see what you are saying with the 35mm, and I said Nikon has lenses as good on the wide end.

The 14-24 reported to be a very good Nikon lens, better than the 14mm though? A close match yes.

Title: Re: DxOMark vs. Reality
Post by: MarioMachado on October 10, 2012, 12:18:18 PM
the DxOMark is just nonsense, no way in this world a Nikon D300S is better than the 7D, even 40D.
Unfortunately where I work (cruise ship), they provide Nikon D300s and the camera sucks big time, poor in every thing. I hate touching this camera!

Canon, the only way!
Title: Re: DxOMark vs. Reality
Post by: marekjoz on October 10, 2012, 12:20:59 PM
Guys, don't you really have enough?
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 10, 2012, 12:25:24 PM
now this was more a answer to the Neuro who finds it difficult to understand that DXO measurements also can be seen / reflected in an image.

Sorry, but what the hell gives you the idea that I don't understand that what DxO measures can be made to show up in images.  Please try to comprehend the following:


So, if you routinely find yourself in situations where you must push shadow exposures 4 stops with less noise impact, you should probably be shooting Nikon (or asking how you manage to get yourself into those situations in the first place..).  If you want to handhold your 600mm lens, you should probably be shooting Canon (or use a 300mm lens with an Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) .  Personally, I don't care what gear you use. 
Title: Re: DxOMark vs. Reality
Post by: Canihaspicture on October 10, 2012, 12:25:42 PM
Then we go into the "shadows" and there  I select an area and let photoshop with auto contrast calculate pixel values​​, and if there are any banding, noise.This is the difference in the lower level = DR
blah blah

So you're saying if as a photographer I constantly screw up exposure I should get a Nikon? With the 5dMk III if I come anywhere near correct exposure within a few stops (which is a huge range) then I get a great photo with little noise if any. With the Nikon even if I absolutely nail exposure I'll get noise at higher ISOs without changing shadows AT ALL...

Correctly expose your shots in camera and get back to us about noise.
Title: Re: DxOMark vs. Reality
Post by: MarkII on October 10, 2012, 01:24:37 PM
Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.
Insulting the original poster while being wrong yourself is not particularly smart.

Of course you can achieve more than the per-pixel DR by downsampling. Suppose two cameras have identical idealised pixel-level DR, but one has double the resolution. With even lighting, the light-per-area is the same on each sensor, but one sensor has double the number of pixels and hence double the headroom. Of course, the higher resolution sensor has more shadow noise, but down-sampling reduces the noise and hence the result is greater DR - more than can be achieved by a single pixel.

In the real world, the light is probably not equally distributed between adjacent pixels, meaning that one may clip before the other (you want something useful from the resolution increase, after-all), and things like system noise also complicate things. However, none of this prevents a down-sampled image having more per-pixel DR than the original sensor pixels, particularly when the downsampling factor is large (4.5x for the D800).

The DXOMark tests are lab-tests, and probably use uniform images for DR (which will tend to maximise the DR benefit from downsampling a D800) and use a lower-noise floor limit that may or may properly account from the pattern noise on many Canon sensors. The DXOMark tests appear to be well performed and accurate - though you need to understand what they are measuring.

If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance. A good example is the 1-bit DAC in some CD players. The DAC is produces a single-bit signal that is hugely oversampled (in the time domain). This is then down-sampled by a filter, yielding a result equivalent to a 16 bit linear DAC (or more).

So not only is it possible to increase DR beyond the native capture range, it is a widely used technique in signal processing.

Of course, the real problem here is that people only want the numbers to show that their purchase is somehow the best and opinion is as valid as understanding. I have seen similar discussions at school in the 1980's - my home computer is better than yours. It is all a bit depressing really.
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on October 10, 2012, 02:23:19 PM
All these posts that say of course you can have greater range on outputs than the sensor is capable of:

On a scale of 1-10, I give you an 11!

<please note the obvious logic flaw and sarcasm>

"But this one goes to 11!"

Or maybe this is a case of "lies, damn lies, and statistics"?  I dunno, it just seems be to be missing the forest for the trees.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 05:11:17 PM
now this was more a answer to the Neuro who finds it difficult to understand that DXO measurements also can be seen / reflected in an image.

Sorry, but what the hell gives you the idea that I don't understand that what DxO measures can be made to show up in images.  Please try to comprehend the following:

  • There are differences between Canon and Nikon sensors
  • Some aspects of Nikon sensor performance are superior to Canon sensor performance
  • Some aspects of Canon sensor performance are superior to Nikon sensor performance
  • DxOMark attempts to quantify those differences and defines their own arbitraty priorities to reduce those measurements to 'Scores'
  • DxOMark Scores are relevant only in specific situations, and those situations may or may not have any relevance for the user of a camera
  • DxOMark Scores attempt to represent a complex system with a single numeric comparison, an effort which is basically doomed to failure from the start, and is ultimately meaningless

So, if you routinely find yourself in situations where you must push shadow exposures 4 stops with less noise impact, you should probably be shooting Nikon (or asking how you manage to get yourself into those situations in the first place..).  If you want to handhold your 600mm lens, you should probably be shooting Canon (or use a 300mm lens with an Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) .  Personally, I don't care what gear you use.

give one example there Canons sensor is better

Here you go;
 
The Canon sensor can fit in the D body with its excelent auto focus and keeper rate,
where as the D800 fits inside a body that has a marginal AF keeper rate.

The Canon sensor can use both the excelent line up of Canon lenses, and with adapter the Nikons as well,
Where the Nikons can not use the Canon lenses.

But, I guess you were talking naked sensors?
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 05:56:35 PM
What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.    ;D

Well, for a start, they run what is by any reasonable measure the leading sensor benchmark site.

They also implemented raw processing software.

How many industry leading benchmarks and raw converters do their critics on camera rumors sites publish ?
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 06:05:21 PM
All these posts that say of course you can have greater range on outputs than the sensor is capable of:

On a scale of 1-10, I give you an 11!

<please note the obvious logic flaw and sarcasm>

"But this one goes to 11!"

Or maybe this is a case of "lies, damn lies, and statistics"?  I dunno, it just seems be to be missing the forest for the trees.

In fact quite the opposite, it is you who are missing the forest for the trees, or in this case, not comprehending that "the sensor" does not consist of a single pixel.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 06:22:45 PM
Are you simple, dishonest or both? The >14-stop-DR fallacy has been discussed multiple times, and has been quite convincingly proven to be just that, a fallacy. You get better SNR from downsampling, but DR cannot magically expand outside the source data boundaries. Get a clue already, please.

I understand that there are people in this forum who repeat the same thing over and over -- that doesn't mean that they have "proven" anything, or even that they are correct.

There are at least two fairly obvious reasons why the DxO mark number is not flawed.

First, the choice of normalization baseline is arbitrary. When comparing (for example) an 21mpx and a 36mpx camera, you could normalize to 21mpx instead and get equivalent (but lower) . You won't hit the "quantization ceiling" if you do this.

Second, even if quantization error is greater than actual sensor error at the black point,the shadow noise level a few EV above the blackpoint will exceed the quantization error. So the bump from 14EV to 14.4EV will manifest as cleaner shadows and  an increase in usable dynamic range.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 06:28:46 PM
Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) . 

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.
Title: Re: DxOMark vs. Reality
Post by: Meh on October 10, 2012, 06:41:05 PM
Oh boy, what a thread.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 06:52:06 PM
What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?

what are DXO´s some would ask.    ;D

Well, for a start, they run what is by any reasonable measure the leading sensor benchmark site.

They also implemented raw processing software.

How many industry leading benchmarks and raw converters do their critics on camera rumors sites publish ?

Many seems to have problems to understand exposure, the subject's DR and the cameras DR when two cameras are compared with the same parameters,   and  also the benefit of a large DR . Throwing sh... on DXO because of DXOs Canon sensor ratings only shows lack of knowledge.

Sure it is important to know if certain cameras have a higher DR, better color or whatever, but none of them are the point in what is wrong with DxO's testing. It is how they arrive at the final number that is flawed and how they present this number to the public.

It has been a while since I consulted DxO and the reviews they put out. The reason is that while they do some reasonable tests that can be informative they make assumptions that the components they feel are important can give a camera an overall rating.  The assume Color Depth is Portrait, DR is landscape and Sports is low ISO. All are important components, and testing individually is good.

If your new to the DSLR world and are trying to learn you flip to the page that shows the D800, it shows a rating of 95. 
You have to take the time to read the fine print to find out that the rating number, as you put it, is just sh.... that DxO is throwing out. There is no standard for combining these three test to arrive at a number, other than the standard DxO has come up with.
Title: Re: DxOMark vs. Reality
Post by: well_dunno on October 10, 2012, 06:52:20 PM
hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
-  They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors  in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

Cheers!
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 06:56:03 PM

So, this discussion, which one has mounted the biggest d1ck... sorry, sensor, in a camera case, is total bu11shit. I think, none of you has ever come into a situation, when he thought stuff like "crap, the dynamic range in this picture is too low. The customer will be very unpleased. I wish, i had a Nikon."


Yes and No

No customer as I am not a Pro Photog

And

Yes, many times a higher DR would have been nice.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 10, 2012, 07:05:01 PM
hmmm, funny how this thread has developed - Neuro's original post was stating DxO has been and is showing scores that suggest sensors in Nikon bodies are better however the reality, that is the economic reality, suggests Canon has been selling more and presumably having better business. Out of this, the logical inferences should be something like the below:

- Those who purchase Canon cams do not look at or do not care about DxO scores
- Those who purchase Canon cams do value DxO scores but other parts of the cameras in which Canon is better are more important to them
-  They are too heavily invested in Canon gear to change brand (this would naturally not explain any increasing lead for Canon) etc etc
if we put the rational choice theory aside the list can be made longer I assume, though I cannot see any discussion about to what extent DxO scores reflect image quality from the sensors  in Neuro's opening. The thread seems to have come off topic...

Those of you who defend DxO scores, why do you have Canon gear?

See my post elsewhere in this thread.

http://www.canonrumors.com/forum/index.php?topic=10020.msg181164#msg181164 (http://www.canonrumors.com/forum/index.php?topic=10020.msg181164#msg181164)

The sensor is an important piece, but it isn't everything. (essentially (2) on your list). I started with Canon because I knew other Canon users. Even then, I was familiar with review sites like snapsort that take DxO scores into account, so Canon needed to deliver a respectable performance.

When I went to upgrade, I could have switched (I had a rebel and two non-L lenses). However, for my application, high ISO performance is much more important than dynamic range at base ISO. So there's a very substantial benefit to having a full frame camera, but as far as I'm concerned, a D800 would not be any better than a 5DIII (indeed, not noticably better than my 5DII) I also think Canon have some compelling advantages over other brands (see my other post)


Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 10, 2012, 07:09:56 PM
give one example there Canons sensor is better

Ok, you like wide DR.  I need to shoot at high ISO.  If I need to shoot at ISO 12800, which sensor offers a wider DR - the D800 or the 5DIII?

do you   understand the benefit and  the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .

Yes, it's one of several ways to extract the maximum amount of information from a RAW file.  But the bottom line is that you can't extract more information than is there to begin with.

Oly OM-D, because DxOMark says it's better than a 7D anyway, right?  ::) ) . 

OM-D sensor could well be better on some measurements (it's a Sony sensor), but if ergonomics with long lenses, and AF performance are a factor, it's a slamdunk for the 7D. Also, since you mentioned shooting with a 300mm lens -- on Canon, you could shoot with the 300mm f/4, the 300mm f/2.8, the 70-300L, the Sigma 120mm-300, or the Canon 100mm-400mm to name a few. I'm not sure what lens you'd use on the olympus.

Which gets back to other points made in this thread -- the system consists of much more than a sensor.

Exactly.  Picking the Civic over the Corvette (was that analogy even in this thread?!?) makes sense if fuel economy or small parking spaces are your biggest needs, picking the Corvette over the Civic makes sense if you engage in street racing, and neither of them make sense if you need to haul your 2-ton yacht to the harbor.

Those of you who defend DxO scores, why do you have Canon gear?

Excellent question! 

Maybe it's a nuance, but speaking for myself, I defend DxO's measurements (usually), but not their scores.  Their scores do not affect my buying decisions because I understand how they generate them (mostly, within the limits of their published methods, which lack some details), and I know how to interpret their underlying raw data for myself and draw my own conclusions based on them.  I also use DxO software for RAW conversions (and I'm pleased as punch that as of today it supports the 1D X, because now I can get away from the kludgy DPP user interface).

Some of the reasons I have Canon gear are that they have lenses that better meet my needs, I prefer the ergonomics of the Canon bodies and menus, and I detest the color yellow.
Title: Re: DxOMark vs. Reality
Post by: Meh on October 10, 2012, 07:19:01 PM
I love Canon cameras and lenses.  There's a certain sexiness to Canon.  Nikon seems so utilitarian... they get the job done but I don't get the same feeling from it.  I need that feeling.  I don't get that feeling from DR at ISO100.  :D
Title: Re: DxOMark vs. Reality
Post by: LetTheRightLensIn on October 10, 2012, 08:18:24 PM

@tnargs, if anything, that just makes the main point of post #1 even stronger.  If DxO truly means that you can only use their scores to compare sensors of similar resolution, that make their results even more meaningless in the real world.  Furthermore, that begs the question - why normalize at all, if you can only compare sensors of simlar resolutions, normalization is moot. 


Exactly.

When I open up Road & Track and compare braking, 1/4 mile, and 0-60, 0-100 for a Ferrari versus a Hyundai, those tests stand up.

When I want to compare CPUs, I can use PassMark to see a plethora of different criteria and I can compare a Intel Celeron M 600Mhz to an Intel Core i7 3960X and QUANTITATIVELY see performance.

Granted in the CPU realm, Motherboard throughput will play a roll, but the speed of the calculations, etc is measurable, definable and COMPARABLE across generations.  So if I can compare the power of a 8 core CPU to a CPU from 8 years ago and measure the difference, how can I not DEFINITIVELY MEASURE a 36 MP sensor to a 10 MP sensor?

If you can't, then the TEST is MEANINGLESS.

The more I read the more I see just how flawed the DxOMarks scores are.  Anyone can produce DATA.  But data is not information. 

What DxOMarks lacks is RELEVANCE

You must be blind then if you can't see the real world difference the extra DR can make in real world properly exposed shots taken with D800 vs 5D3. It's plenty relevant for some shots.

Title: Re: DxOMark vs. Reality
Post by: LetTheRightLensIn on October 10, 2012, 08:19:21 PM
DxOMarks are useless and only made to entertain hobby photographers.

Yay! I love to run around bragging about how ignorant I am! Yay!
Title: Re: DxOMark vs. Reality
Post by: LetTheRightLensIn on October 10, 2012, 08:29:02 PM
I do agree that it's best to look at their plots and RAW numbers and not there overall scores since they may not weigh various factors the way you would plus there isn't really any a way to look at so many different aspects at once and come up with one universal score to give meaning to it all at once.

I also agree that their lens testing appears to leave MUCH to be desired. Lens testing is done by different people there and it's much trickier to carry out well. I ignore their lens tests but their sensor data generally seems to be pretty solid.

OK maybe DxO tests have not crashed the market for Canon yet, but as you say they, and general more over the top crippling of silly little things, sure have made their user base restless as you can tell but peeking at any forum on the web. Switching systems is a big deal for many (it's a pain, costs some money, some don't like the Nikon UI as much, Nikon doesn't have 70-200 f/4 IS and such they are mostly all huge fast stuff or consumer stuff) and it can take a while for things to build up to make many switch, maybe they did not get there quite yet, but they could be heading there. Also the bulk of sales are probably in the Rebel market. Also the 5D2 was aided a lot by the video guys nabbing them like mad and Nikon hasn't had Exmor in FF smaller, more reasonably priced bodies until recently, etc.

And whoever markets things better often matters even more. Apple IIe and IBM PC were utter trash as was Windows for a long time (and even now to an extent) and yet that stuff sold and the far more advanced stuff, sadly, did not thanks to horrible management and a computer press often too easily bought and paid for.
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 10, 2012, 09:00:13 PM
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.
Title: Re: DxOMark vs. Reality
Post by: Meh on October 10, 2012, 10:12:39 PM
Oh boy, what a thread.

yes and mybe some have learn something
1 there is noting like a right exposure
2.How you expose depends on the motive and the  motives DR and what camera you use
3.With a camera and 14 stop DR you can cover more from the topp to the bottom, and adjust with curves and other tools and  even mark an area and do a selektive adjustments to show more information from shadows  for example.

It's a nice thought you have that somebody might have learned something... but sadly the evidence suggests that somebody hasn't learned a thing.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 10, 2012, 11:09:31 PM
At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800  red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data? 
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 10, 2012, 11:09:42 PM
give one example there Canons sensor is better

Granted, the difference above is pretty small.  So, while we're at it, let's compare the Canon 7D with it's contemporary competition, the D300s (released about one month apart).  Which has better DR through almost the entire ISO range, according to the almighty DxO's scoring and normalized data?
Title: Re: DxOMark vs. Reality
Post by: tnargs on October 11, 2012, 02:19:58 AM
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.

Makes one wonder if he is just a troll.... posts are exclusively anti-Canon pro-Nikon (Sony type N)
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 11, 2012, 03:30:21 AM
Here we go again with the same discredited arguments ...

I, for one, can't take DxOMark seriously or trust any of their numbers when they...

* Rank $40,000 medium format digital backs lower than consumer APS-C DSLRs.

There's nothing necessarily "wrong" with this per se. A medium format back is not necessarily better as a general purpose camera than an APS-C camera.

You know darn well their scores do not reflect how good the camera is as a "general purpose camera."
Quote
I doubt that anyone is seriously using the website to decide whether to choose an APS-C or a MF back, so this argument is a silly red herring (usually trotted out by camera "fans" of low scoring cameras)

It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras. When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede. When the same "fan" is confronted with the fact that the score for a consumer DSLR is higher than a 40 MP MF back, all of a sudden the score isn't a score of overall IQ, and nobody is using those scores to begin with, and stop tossing out red herrings.

Which is it?

Quote
This horse has been beaten to dust. They report 13.2 bits for each pixel. You can gain dynamic range by downsampling. 14.4 stops is based on downsampling to 8mpx.

You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

Quote
Quote
* Report values for dynamic range that I know to be false from both personal experience and testing. (They rank the 10D, 20D, and 7D about the same. The 7D is a good 2 stops better.)

You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

Quote
For all the critics of DxOMark critics, I would like to point out that no less a professional and respected figure than Michael Reichmann stopped using DxOMark because of the obvious errors he observed in their results.

What precisely are his criticisms ? What exactly are his credentials as far as engineering and benchmarking are concerned ?
[/quote]

Visit his website.
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 11, 2012, 03:39:27 AM
Ok, I take this as asking:

Quote
Nonlinearity has not yet been incorporated into sensors. You can compress more than 8 stops into an 8-bit JPEG this way (i.e. Canon HTP), but RAWs are simply not non-linear at this time.

Get a clue man, you have no idea what you are talking about. Sensors and amplifiers are nonlinear, whatever you do.

I love how the excuse for DxO's results keeps changing. It's due to down scaling. No, it's due to non-linear sensors / amps. No, it's due to... You guys need to get together and compare your notes   ::)

The sensors are not designed to expand DR through non-linear behavior. They are, for the purposes of this discussion, linear. Further more, they have noise. And like all analog to digital systems their range is actually less than on paper because of things like noise.

If they were absolutely perfect in every way you could expect a maximum of 14 stops. They're not.

Face it: DxO botched another test.

Now if you'll excuse me, I have to order my Nikon D3200 through Amazon since it's better than a Hasselblad. Talk about a steal  ;D
Title: Re: DxOMark vs. Reality
Post by: dtaylor on October 11, 2012, 03:49:55 AM
If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.
Title: Re: DxOMark vs. Reality
Post by: MarkII on October 11, 2012, 04:54:02 AM
If you still do not accept that it is possible to increase DR by down-sampling, I would take a look at the many applications in which oversampling is used to improve performance.

Over sampling and down sampling are two different things.

Yes - but possibly not in the way I think you mean. The original RAW image is spatially oversampled (20MP or more) with respect to the final output resolution (8MP). The process of converting the original high-resolution image to the smaller image is downsampling.

The Wikipedia pages are helpful here, and explain this probably better than I can:

  http://en.wikipedia.org/wiki/Oversampling (http://en.wikipedia.org/wiki/Oversampling)
  http://en.wikipedia.org/wiki/Downsampling (http://en.wikipedia.org/wiki/Downsampling)

These also give an example of how this can be exploited to give more resolution (DR) than the actual converter (sensor pixel).
Title: Re: DxOMark vs. Reality
Post by: tron on October 11, 2012, 06:36:04 AM
do you   understand the benefit and  the choiche of underexposure and moving middle grey down 1-4 levels and bring in highligts far above 3.5 stops .
This is interesting! I have also read it somewhere in this forum before and I was puzzled about one thing.

So please allow me to ask something:

If the middle grey is moved down 1-4 levels and the highlights are moved above then don't we lose the
mid tones?

Title: Re: DxOMark vs. Reality
Post by: elflord on October 11, 2012, 06:40:02 AM
Quote
It's not relevant whether or not people are looking to them, and this is not a red herring. Their scores are consistently presented by them and others as IQ scores, yet their methodology is obviously flawed if they can give a consumer APS-C sensor a higher IQ score than a MFDB. This obvious error is usually ignored by "fans" of high scoring cameras.

It is a red herring, because it is not a "flaw" in methodology, it is a limitation in using a single score to describe "overall" performance.

It does not follow at all that the measurements are in any way incorrect.

As a line of argument, it is, in essence, an ad-hominem -- an attempt to troll the data for "incorrect" results, and attack the source, instead of addressing the real issue (which is that Canon's sensors don't perform well). Besides being a morally reprihensible approach, it is a logical fallacy. It suffers from all the dangers inherent in data mining (that if you look for a given result over a large amount of data, you can usually find it, especially if you don't have a very good understanding of what you are looking at).

Quote
When a "fan" wishes to compare, say, Canon and Nikon, then all of a sudden the score is a score of overall IQ that we all must concede.

No, the score represents some subjective weighting of different measurements. "Overall" image quality is also a subjective notion (what does "overall" mean ?)

The measurements however are cold hard factual data. And in the case of Canon vs Nikon, they show that Canon struggles with shadow noise at low ISO. If you often use the camera at ISO 800 or more, this may not matter a whole lot, but if you shoot at ISO100 all the time, it might. The results put forth by way of DxOs measurements (again specifically shadow noise) have been validated by other users/commentators, so no amount of carrying on about medium format and other red herrings will make this go away.

You cannot gain dynamic range by down sampling because you throw away detail with noise. If you think you can, your definition of DR is flawed.

Sure you can. Dynamic range is the space between saturation point and a reference SNR. If your reference SNR is 0db (SNR=1), then quantization will bite you. But if you use a different floor, it won't. In practice, SNR=1 isn't where usable dynamic range starts, so you do gain by going from "14" to "14.4". Of course there is a trade between "detail" (spatial resolution)  for noise reduction -- that's why they normalize to the same resolution, it's also why it doesn't make sense to compare a high resolution to low resolution camera without doing that. BTW, some of these medium format (e.g.  80mpx )  backs you are fond of would not look nearly as good if you insisted on using a per-pixel metric to measure their performance.

Also, see my other point -- if you compare two cameras one at 21mpx and one at 36mpx, you could normalize to 21mpx instead of 36mpx. The choice of "target resolution" does not matter, the important point is that everything is normalized to the same resolution.

Quote
Quote
You keep saying that these are "about the same", and I keep calling you on it. They are not "about the same".

They show a 0.7 stop difference when it's roughly 2 stops.

However you slice and dice it, if you look at the two graphs, it's not "about the same". It's 0.7 stops on the Y axis at base ISO, or about 2 stops on the x axis (meaning that I can crank ISO up two stops on the 7D and get the same DR). That's a very substantial technological bump.

Quote
Visit his website.

I did, and my question stands -- I saw information on what he has done as a photographer. Again, as a benchmarker, engineer, etc, what are they ?
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 11, 2012, 07:52:15 AM
This is 5dmk2 and d800 shooten against sun and clouds, exposed so the sky and clouds are visible,
same time, f-stop and 100iso
both images  lifted  in the same way in photoshop so we should be able to see country landscape.

DR   

Sorry but this set is bull. Not a believable comparison at all.

Are they, explain why.
Both cameras equal exposed  time/f-stop / iso  so there will be no highligts clipping, both cameras equaly lifted in the shadows =visible results

1, You are showing two pictures that are JPG without any explanation of the full work flow you used to arrive at the results.
2, You modified the pictures in photoshop. No explanation of what format you used, or if you did any processing on the RAW files.
3, I have shot similar sunsets with the 5D II. The results you are proposing are not what I would reasonably expect to see.

Last, this appears to be an attempt to exaggerate results to try and show the DR of D800. The D800 was measured by DxO. DxO's own results were not as wide as you are trying to make out.

Granted if your point is that the D800 has wider DR, it does. DxO measured it at over 14, but you do not have 14 stops of usable DR. You have a sensor that was measured at the extremes to have 14 stops. In reality you have far less. The same is true of the 5D II. What was measured as at extremes for both to be a difference of 3 stops, if DxO had tested what a camera can actually do in real world situations the difference would be closer to 1 or 2 stops. Still this is a fair difference, but to keep beating the 14 stop drum and warping pictures to over exaggerate a point that most here would concede to forces one to question your points.

The first set of pictures you posted demonstrated the point, and now I wonder if they were accurate. I have seen the side by side comparisons from reasonable sources and saw no need to scrutinize closely your first example because the difference is there and has been well established by other sources. But on the second it makes me question and doubt your examples.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on October 11, 2012, 08:14:31 AM
At 6400 they are equal, at 12800 I need to rezise the d800 size to 5dmk3 size and they are very similar, above 12800 there is small advatage to 5dmk3 BUT with little more NR they look both OK
As you can se here blue d800  red 5dmk3

Wait, I thought we were talking about DxO's oh-so-reliable-even-with-normalization data?

select print mode or make d800 pixel number  as many as Canon

LOL.   You're really grasping at straws now...  Go look again (http://www.canonrumors.com/forum/index.php?topic=10020.msg181644#msg181644), but more carefully this time.  I did select Print mode. 

I'll even make it easy for you by reposting it, along with the Screen mode where as you can see (or can you, with your blinders on?) that the 5DIII has a greater DR advantage at high ISO without normalization. 
Title: Re: DxOMark vs. Reality
Post by: marekjoz on October 11, 2012, 08:44:48 AM
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...
Title: Re: DxOMark vs. Reality
Post by: marinien on October 11, 2012, 09:10:19 AM
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.
Title: Re: DxOMark vs. Reality
Post by: marekjoz on October 11, 2012, 09:14:23 AM
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.

<sarcasm off> :D

Anyway - the bigger the area in either low or high iso part of the graph, the bigger the difference between them there and the simplest way to calculate this area are integrals when you know the math formulas. But anyway it was just to cheer you all up in this serious discussion...
Title: Re: DxOMark vs. Reality
Post by: mystic_theory on October 11, 2012, 09:25:23 AM
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality?  ;D
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 11, 2012, 09:44:35 AM
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality?  ;D

No, it means that the 1D X tested well with the three tests that DxO uses to generate their make believe scores.

Title: Re: DxOMark vs. Reality
Post by: Alrik89 on October 11, 2012, 10:02:50 AM
Guys, you've got a real problem, when you assess camera sensors basing on some stupid numbers like DxO.

If you're a professional photographer (a skilled one) then none of your customers will ever see the difference in the IQ/dynamic range/whatever between a Nikon and a Canon DSLR.

So, this discussion, which one has mounted the biggest d1ck... sorry, sensor, in a camera case, is total bu11shit. I think, none of you has ever come into a situation, when he thought stuff like "crap, the dynamic range in this picture is too low. The customer will be very unpleased. I wish, i had a Nikon."

Get a life.

A skilled photographer is not the same as  he or she  is a good  copyist and know how a digital work flou works.  Have you  anytime worked in a darkroom with a negative film  then you will understand what I mean and likewise it is with a raw files.

A skilled photographer with also skills in the digital "dark room" kan do so much more than a photographer with no  photoshops skills.  And even more with a camera who has 14 stops DR.
best regards , pro since 30 years back.

Well, when you need 14 stops DR to "rescue" your pictures via Photoshop, then you should ask yourself, if you`re a real pro.
When it comes to the the point where you really need 14 stops of DR, you've taken your picture in a wrong way.

And: none will see the difference. Neither a customer nor other pro photographer.
Title: Re: DxOMark vs. Reality
Post by: marinien on October 11, 2012, 10:22:17 AM
Can someone please calculate and compare the surface areas between the orange and red lines because I'm still not convinced when and how to interprete those graphs and which sensor is better and when?
DxO measurements is a bullshit if they can't provide the simple math formulas to calculate the integrals. I'm so dissapointed...

Euh, I don't see any reason to calculate the surface areas between the lines in order to know which sensor is better and when.
Just have a look at theses graphs again and forget everything about integral.

<sarcasm off> :D

Anyway - the bigger the area in either low or high iso part of the graph, the bigger the difference between them there and the simplest way to calculate this area are integrals when you know the math formulas. But anyway it was just to cheer you all up in this serious discussion...

Haha, thanks  ;)
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 11, 2012, 12:22:26 PM
The 1DX got good grades from DxO: does that mean it's a crappy camera since those grades have nothing to do with reality?  ;D

This will be very intresting to know, Canon have increased the DR   and shows very good high iso scores
if the figures are OK  ( are  DXO still crap?)

Yes, the assumptions and overall score they draw from the tests are still crap. There scores are drawn from to narrow of a parameter to be of real use.
Title: Re: DxOMark vs. Reality
Post by: shinyknights on October 11, 2012, 03:01:53 PM
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III)
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800 (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800)

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml (http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml)

Any thoughts on this guys?
Title: Re: DxOMark vs. Reality
Post by: PackLight on October 11, 2012, 03:17:58 PM
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III)
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800 (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800)

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml (http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml)

Any thoughts on this guys?

I think if you use DxO information it is important to know in what context they test, the test reults and ignore the values they place as scores.

As for the test at ISO 12800 you mention, as far as I am concerned the samples would be irrelevant as well because I wouldn't go that far with the 5D. I would with the 1D X.  I think a comparison at ISO 6400, an ISO I might go to on occasion tells more as does the ISO 3200 comparison.
Title: Re: DxOMark vs. Reality
Post by: K-amps on October 11, 2012, 04:24:19 PM
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Canon/EOS-5D-Mark-III)
http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800 (http://www.dxomark.com/index.php/Cameras/Camera-Sensor-Database/Nikon/D800)

And I'm reading more and I see examples on cameralabs.com.
And here, they show that Canon's 5D Mark III destroys Nikon's D800, particularly at 12800 ISO.

http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml (http://www.cameralabs.com/reviews/Canon_EOS_5D_Mark_III/Canon_5D3_vs_Nikon_D800_noise.shtml)

Any thoughts on this guys?

They have certain criteria like 30db signal to noise ratio ... any shot that gets below this is tossed out. If they change their criteria to (taking an arbitrary number) 25db, then perhaps 5d3 will have 11000 ISO (another arbitrary number)  and D800 will have 4000 ISO (Yet another arbitrary number)... shifting the criteria around will swing the score back to the Canon Camp. Thats what pissed people about them is the criteria they use to weigh qualities of a sensor for it's "Overall Score".

The criteria they use effectively nullifies Canon's real world high ISO advantage.

The other criteria they have in place is that at the 30db SNR level, at least 18 bit color depth be mantained for the ISO score, now who plucked this number from someones...  expletive is another debate altogether, but it again weights the balance in favor of the Sonikon sensors... (The color bit depth rears it's ugly head again for overall scores.)

I think even with current technology levels fo Canon sensors, with their high ISO advantage, DxO could tweak their criteria and Canon scores could be above Nikons... but there is one area where playing fuzzy math will not help canon and that is the DR number... this is what many Canon Loyaists want Canon to improve.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 11, 2012, 08:00:28 PM
I'm curious if DxOMark is a credible source of info.
Here, they say that the Canon 5D Mark III is good until 2293 ISO
With Nikon's D800, it says that it is good until 2853 ISO.
They dont even provide picture samples!

Their scores are based on measurements, not picture samples.

If you want the full story behind the scores, click on the measurements tab. The scores compress a lot of information using subjective (and therefore somewhat arbitrary) criteria.
Title: Re: DxOMark vs. Reality
Post by: straub on October 12, 2012, 03:29:33 AM
Yes - but possibly not in the way I think you mean. The original RAW image is spatially oversampled (20MP or more) with respect to the final output resolution (8MP). The process of converting the original high-resolution image to the smaller image is downsampling.

The Wikipedia pages are helpful here, and explain this probably better than I can:

  http://en.wikipedia.org/wiki/Oversampling (http://en.wikipedia.org/wiki/Oversampling)
  http://en.wikipedia.org/wiki/Downsampling (http://en.wikipedia.org/wiki/Downsampling)

These also give an example of how this can be exploited to give more resolution (DR) than the actual converter (sensor pixel).
And that must be why an 8KHz downsampling of a CD has more than double the dynamic range of the original, right? You can google theorems, fine, but apparently have no idea what they mean and where they are applicable. So you must work at DXO, then?

Try a test--take a D800 photo of Stouffer wedges, push exposure in post how much you like, and make a note of the darkest one. Now downsample to 1/64, i.e. ~0.5mpix and repeat. How many extra wedges appeared? Exactly as many as how much the *oversampling theorem* is applicable here--i.e. zero.
Title: Re: DxOMark vs. Reality
Post by: MarkII on October 12, 2012, 05:46:57 AM
And that must be why an 8KHz downsampling of a CD has more than double the dynamic range of the original, right? You can google theorems, fine, but apparently have no idea what they mean and where they are applicable. So you must work at DXO, then?
The downsampling will reduce the bandwidth but increase the resolution of the samples. You trade off precision in the amplitude (quantisation noise) for bandwidth (loosing high frequencies). This is exactly the same as happens when you down sample a digital image - and just as you might not want to listen to all your CDs at 8KHz you probably don't want to print out all your images at 600x400 no matter how much this improves the DR.

I posted the Google links as they are usefully informative on the subject. You, rather than posting an informed and referenced response, just posted an insult (FYI, I am a Physicist  who has worked with DSP and signal processing for more than 20 years, and I have no connection to DXO).

Try a test--take a D800 photo of Stouffer wedges, push exposure in post how much you like, and make a note of the darkest one. Now downsample to 1/64, i.e. ~0.5mpix and repeat. How many extra wedges appeared? Exactly as many as how much the *oversampling theorem* is applicable here--i.e. zero.
Well, I do not have a D800 to test (and have not commented on this specifically).

I find that the DXO measurement data matches pretty well my experience when shooting with those cameras that I have (GF1, 40D, 5DII, 5DII). You can take issue with the fairly arbitrary criteria that they use to produce their overall scores and I am sure that DXO sometimes make mistakes, but the individual measurements - if you understand what they are - have always looked pretty good to me and correlate well with my experience of the 5DIII (particularly when compared to the 5DII).

Simply wanting something to be different does not make it so.
Title: Re: DxOMark vs. Reality
Post by: tron on October 12, 2012, 06:46:51 AM
I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when  there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Thanks.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 12, 2012, 07:25:17 AM
I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when  there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.

Title: Re: DxOMark vs. Reality
Post by: tron on October 12, 2012, 09:48:57 AM
I can understand the improvement in noise when downsampling but can someone prove that downsampling an image (to 8Mpix for example) improves DR?

I ask this because I believe in practice what is burned in the highlights has been lost forever and when  there are shadow areas I cannot image how a "dark" pixel will benefit from its neighbour equally dark ones.

Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.
Thanks!
Title: Re: DxOMark vs. Reality
Post by: nightbreath on October 13, 2012, 02:44:08 AM
Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.
I didn't quite catch this. Could you expand your answer on my example? Here's how I understand downsampling:

Assume that we have 4 neighboring pixels that represent in reality 4 black squares, but in our RAW file (due to noise impact) they have next brightness levels:
1 0
0 3

So, when I downsample I get 1 pixel that has brightness level 1 (for example, if I downsample by next formula: (1+0+0+3)/4 ).

It means that before downsampling we were looking at image with average noise level equal to 1 and we do the same after downsampling.

So what am I doing wrong compared to all of you that have noise level reduced after downsammpling?

P.S. Sorry for asking silly questions, i'd just like to understand  :)
Title: Re: DxOMark vs. Reality
Post by: BXL on October 13, 2012, 06:38:39 AM
Those of you who defend DxO scores, why do you have Canon gear?
Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks" ;)
Title: Re: DxOMark vs. Reality
Post by: heptagon on October 13, 2012, 07:43:39 AM
Those of you who defend DxO scores, why do you have Canon gear?
Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks" ;)
You wouldn't go offroad with a racecar or wear slick tires in the mud.

Every job needs the right tool.
Title: Re: DxOMark vs. Reality
Post by: elflord on October 13, 2012, 10:18:40 AM
Dynamic range is the number of stops between "saturation point" and blackpoint. Saturation point is where the highlights get "burned out". This doesn't change. Black point is the point at which SNR is 0db (that is, signal to noise ratio is 1).  Downsampling reduces noise, so the SNR at what used to be the blackpoint goes up (to 5db for example) and the new blackpoint after downsampling  is some way down from that.
I didn't quite catch this. Could you expand your answer on my example? Here's how I understand downsampling:

Assume that we have 4 neighboring pixels that represent in reality 4 black squares, but in our RAW file (due to noise impact) they have next brightness levels:
1 0
0 3

So, when I downsample I get 1 pixel that has brightness level 1 (for example, if I downsample by next formula: (1+0+0+3)/4 ).

It means that before downsampling we were looking at image with average noise level equal to 1 and we do the same after downsampling.

So what am I doing wrong compared to all of you that have noise level reduced after downsammpling?

P.S. Sorry for asking silly questions, i'd just like to understand  :)

There are a couple of issues with the above example. First, the right measure for error is RMS, not arithmetic mean (that is, if you take two measurements, +1 and -1 for a variable whose true value is 0, the "error" is certainly not 0). 
In your example, the RMS error is

sqrt( 1/4( 32 + 12 + 02 + 02) ) = 1.6.

 After you downsample, you get an RMS error of 1 -- that is, sqrt( (1-0)2/ 1 ).

Second the example is a bit unusual because you generally don't expect the error terms to be entirely of the same sign, and by averaging the signal the positive and negative errors cancel each other out.

If you have (for example) a "true" value of 10 and readings 6,3,9,17 (these I just randomly sampled from a normal distribution with mean 10, standard deviation 5).  We average these and we get 9 (an error of 1), whereas before we had an error of 5 -- 

sqrt[  ((10-6)2+(10-3)2)+(10-9)2+(10-17)2))/4  ].

In general, the expected value of our error drops by a factor of sqrt(N) -- so in my above example, the expected error in the beginning is the standard deviation of our distribution, 5. After we average four pixels, our expected error is 2.5  (5/sqrt(4)).
Title: Re: DxOMark vs. Reality
Post by: well_dunno on October 13, 2012, 08:05:27 PM
 :) It was more of a rhetorical question, which was the last line of my post earlier in this thread...

Cheers!
Title: Re: DxOMark vs. Reality
Post by: nightbreath on October 14, 2012, 10:22:33 AM
There are a couple of issues with the above example. First, the right measure for error is RMS, not arithmetic mean (that is, if you take two measurements, +1 and -1 for a variable whose true value is 0, the "error" is certainly not 0). 
In your example, the RMS error is

sqrt( 1/4( 32 + 12 + 02 + 02) ) = 1.6.

 After you downsample, you get an RMS error of 1 -- that is, sqrt( (1-0)2/ 1 ).

Second the example is a bit unusual because you generally don't expect the error terms to be entirely of the same sign, and by averaging the signal the positive and negative errors cancel each other out.

If you have (for example) a "true" value of 10 and readings 6,3,9,17 (these I just randomly sampled from a normal distribution with mean 10, standard deviation 5).  We average these and we get 9 (an error of 1), whereas before we had an error of 5 -- 

sqrt[  ((10-6)2+(10-3)2)+(10-9)2+(10-17)2))/4  ].

In general, the expected value of our error drops by a factor of sqrt(N) -- so in my above example, the expected error in the beginning is the standard deviation of our distribution, 5. After we average four pixels, our expected error is 2.5  (5/sqrt(4)).
Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.
Title: Re: DxOMark vs. Reality
Post by: bdunbar79 on October 14, 2012, 11:07:30 PM
Those of you who defend DxO scores, why do you have Canon gear?
Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks" ;)

    There are different brand fanatic  and the arguments regarding why something suddenly not important anymore (when "their" brand suddenly is not the best ) and this  might take quite comical proportions ... Some very active writers, among others  here at Canonrumors's  are direct rabid - this despite the notice that they are intelligent enough to see the connections, there is an instinctive religious barrier that prevents them to see what's right in front of their eyes ... :-)

Oh the irony......................
Title: Re: DxOMark vs. Reality
Post by: birtembuk on October 15, 2012, 01:28:52 AM
Those of you who defend DxO scores, why do you have Canon gear?
Because Canon offers the better package. It's not all about the sensor. It's the combination of the sensor, AF, ergonomics, processor and lenses. Similar to cars, where one company offers the better engine, but another company offers the better car. The best engine won't do the job if the car "sucks" ;)

    There are different brand fanatic  and the arguments regarding why something suddenly not important anymore (when "their" brand suddenly is not the best ) and this  might take quite comical proportions ... Some very active writers, among others  here at Canonrumors's  are direct rabid - this despite the notice that they are intelligent enough to see the connections, there is an instinctive religious barrier that prevents them to see what's right in front of their eyes ... :-)

Religion ? What are you talking about mate ? To the vast majority, the biggest size they'll go is the 23" diagonal of their computer screen. Who is printing billboards ? If the dxo ratings were saying that my Canon gear is better than the competition, I for one would not give a damn. Conversely, I don't give a damn with the opposite. If your purchase decision is based on dxo ratings, please be my guest. But maybe one day you'll have to dump all your gear because dxo found out that the iPhone has a much better sensor ! For sure, dxo won't make me change a micron from my path. Religion or not.       
Title: Re: DxOMark vs. Reality
Post by: MarkII on October 15, 2012, 02:07:47 AM
Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.
I think you should explain how you conclude this from Elfiord's post - it does not follow at all.

The down-sampled figures are a perfectly reasonable way of comparing sensors. They correspond to what you can achieve if you print or view your photograph out at the same physical size and print DPI. From the numbers, it looks as if DXO actually measure this after performing down-sampling, rather than just applying a theoretical perfect adjustment. This means that the down-sampled numbers inherently include any 'environmental parameters' in the result.

No one argued against downsampling when the comparison was 5DII to D700...
Title: Re: DxOMark vs. Reality
Post by: nightbreath on October 15, 2012, 06:24:25 AM
Got it now. So it pretty much describes why you cannot normalize without taking into account environmental parameters. DXO is doing something similar to comparing a bee and an airplane from bee's perspective. In the end we just get artificial results not possible in real world.
I think you should explain how you conclude this from Elfiord's post - it does not follow at all.

The down-sampled figures are a perfectly reasonable way of comparing sensors. They correspond to what you can achieve if you print or view your photograph out at the same physical size and print DPI. From the numbers, it looks as if DXO actually measure this after performing down-sampling, rather than just applying a theoretical perfect adjustment. This means that the down-sampled numbers inherently include any 'environmental parameters' in the result.

No one argued against downsampling when the comparison was 5DII to D700...
I was not brand-specific and I'm not defending 5D Mk II vs. D700 scorings. As well as others I didn't have a clue what to look at when I first got to the DXO web-site and wanted to choose my next camera. That's why I am kind of frustrated now, because I kno what to look for.

As for environmental parameters, I was referring to ADC bit depth, as it is clear that by downsampling you just create data that was not in the image before downsampling (more than 14 stops of data in 14 bit depth image). I.e. if your measurements include noise floor evaluation that gives different results when you compare current image with other images, you just won't make an impression of a trusted source.

Why not come up with an algorithm that measures the whole image, not each pixel individually? In this case you don't need to downscale at all.

P.S. Please correct me where I'm wrong  ;)
Title: Re: DxOMark vs. Reality
Post by: aj1575 on December 13, 2012, 08:44:09 AM
Please look at these graphs:

And now tell me, which sensor should get a better mark for high ISO performance?
Title: Re: DxOMark vs. Reality
Post by: marekjoz on December 14, 2012, 03:33:42 AM
Please look at these graphs:

And now tell me, which sensor should get a better mark for high ISO performance?

The Nikon's one? :)
Title: Re: DxOMark vs. Reality
Post by: infared on December 15, 2012, 09:49:45 AM
The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data.  Sometimes thier "data" is in line with the general consensus of my other respected sources, many times not. I do know that specific products from manufacturers can vary from item to item..but DxO mark does not present their information with that variation. They present their information as "fact".
 I do not waste my time with ANYTHING that DxO Mark has to say because I have found many of their "opinions" to be less than valuable. I feel that anyone with experience and intelligence would do the same.
There are just so many other consistently reliable sources out there to get my information from that increase my awareness and knowledge of photographic tools.
Title: Re: DxOMark vs. Reality
Post by: Northstar on December 15, 2012, 10:27:29 AM
I occasionally  read their reviews but take them with a huge grain of salt....they continue to do things that make me think they are unorganized and don't have their "stuff" together.   I have a few other reviewers that I prefer.

The latest thing I noticed was when I checked to see if they had done a review on the 135L, this was in September.  They didn't have a review but they did have a note saying they would be posting a review of the 135L in October.  So I check back in November, no review, but a note saying they'll be reviewing it in November...So I forgot about it until now, I just checked, they still haven't reviewed the lens, and the note has been changed to...(copied from their website)

This product will be tested and reviewed on December on dxomark.com. Stay tuned by subscribing to our newsletter.

So it's December 15th and no review...anyone want to bet that it won't be done in December?  Where I come from, if you say one thing and do another, again, and again....it just isn't good, you lose credibility.

Also, another typo where they write "on December".  Their website is full of typos. 

IMO, they are overrated as a "review" website.



Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 12:02:18 PM
The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data. 

DXOmark are THE industry leader in sensor benchmarking. There is nothing "suspect" about their sensor measurements. The minutiae of sensor benchmarking has been debated here, and the overwhelming conclusion of the discussions is that the measurements for their sensor benchmarks are sound. There is some nitpicking about the way those measurements are aggregated but that's about it.

I personally would hold off buying a camera body until DxO have tested it. I don't have nearly as much confidence in other sources for testing sensors.

Lenses are a separate issue, there are other sources that do a better job of testing lenses.
Title: Re: DxOMark vs. Reality
Post by: Zlatko on December 15, 2012, 12:31:38 PM
The thread's first post has a lot of unproven speculation....but I do not disagree with the basic sentiment. DxO is NOT a place this photographer would go for meaningful information when researching cameras and lenses for purchase..  I do not know the reason for their wild fluctuations from what I know to be reality, but they are what they are....and to me what they are is a SUSPECT source, that often has stated opinions that are not in line with the conclusions of a pool of other reliable and consistent sources. DxO Mark backs up their remarks, in my opinion, with suspect "scientific" data. 
DXOmark are THE industry leader in sensor benchmarking. There is nothing "suspect" about their sensor measurements. The minutiae of sensor benchmarking has been debated here, and the overwhelming conclusion of the discussions is that the measurements for their sensor benchmarks are sound. There is some nitpicking about the way those measurements are aggregated but that's about it.

I personally would hold off buying a camera body until DxO have tested it. I don't have nearly as much confidence in other sources for testing sensors.

According to this article on Luminous Landscape today —
http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml (http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml)
— the  overall Camera Sensor score is biased toward single shot HDR at low ISO settings — a capability we never had in the past and which we may infrequently need.

So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

We also learn that their Portrait score is misnamed.  "Essentially it measures choma noise in the dark parts of a low-ISO image" — which may be relevant to some photographers, and not at all relevant to others.  This explains why two cameras can have the same Portrait scores and yet one can be clearly better for actual portraits.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 12:55:19 PM
So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 15, 2012, 01:08:54 PM
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)
Title: Re: DxOMark vs. Reality
Post by: Zlatko on December 15, 2012, 01:16:21 PM
So they may be the industry leader, and they may be scientific, but it seems their big headline-grabbing Camera Sensor score may not be very meaningful to a lot of photography.

This would be true of any single score. If they gave all the weight to high ISO performance, you would see medium format backs getting lower scores than point and shoots.

Thankfully, they don't just publish the single score -- they publish the three use case scores, and all the measurements. Their website also makes it easy to plot measurements of two different cameras on the same axes, so that when a new camera gets a surprisingly high or low score, it's easy to determine why.

The luminous landscape article is overwhelmingly positive. If the most serious criticism is nitpicking over choices of naming, that's a pretty positive review.
Then photographers would be correct to ignore the Overall Score.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 01:25:45 PM
Then photographers would be correct to ignore the Overall Score.

If you care about sensor performance, you should pay attention to DxO's measurements because no-one else is going to do a better job at benchmarking sensor performance. Not the overall score, but the use case scores and the graphs.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 01:47:36 PM
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.
Title: Re: DxOMark vs. Reality
Post by: Zlatko on December 15, 2012, 02:01:45 PM
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)

I agree that trying to condense everything into a single number is problematic, and you would have a similar problem with cars -- which is "better", a ferrari 458, a toyota prius  or a ford explorer ?

What I don't see is what the analogous cameras to the above cars in your example would be. Perhaps medium format vs full frame (with the former getting low numbers because of limited ISO capability) but the MF back really is analogous to a ferrari -- it is designed with a particular purpose in mind, and therefore if you are in the market for that product you probably aren't very interested in benchmarks that force it to play on everyone else's terms.

But I don't see for example the difference between the various mainstream APS-C and full frame SLR sensors to be analogous to your example.
What is analogous is that his example condenses everything into a single number, which you agree is problematic.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 15, 2012, 02:03:43 PM
The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless.  But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores.  Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.
Title: Re: DxOMark vs. Reality
Post by: Northstar on December 15, 2012, 02:39:13 PM
Quote
They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 02:45:36 PM
The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.

To me what seems to be driving the outrage here is that Canon sensors get lower DxO scores than Sony sensors. But the relative rankings here are pretty well deserved -- Canon gets trounced at base ISO. It's not even close. At high ISOs, sometimes the Canon sensor inches ahead but the gap is small. So while it's true that there are many shooting scenarios where the Canon sensor is equal or even slightly better than the Sony, if you had to honestly answer the question, who makes the better sensor  and had to give a straightforward answer without going into the nuances of use cases, could you say "Canon" with a straight face ?

Quote
But how many people look at the Measurements?  I suspect it's much more common for people to just stop at the scores. 

This is probably true, and if they are just looking at the scores, they are apparently not interested in digging any deeper than that.

Quote
Furthermore, DxO's scores are also reported by other sites, e.g. Snapsort, without reference to the underlying measurements.  They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

I don't completely agree with this. The "high ISO score" already takes into account performance at higher ISOs. This score is a pretty good indicator of performance for those who do most of their shooting above base ISO. The dynamic range score is a pretty good indicator if someone is shooting at base ISO.

Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 15, 2012, 03:03:35 PM
Quote
They report DR, for example, but there's no mention that it only applies at base ISO.  That original DxO bias is propagated elsewhere.

Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number?   I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

again...continue to become less and less impressed with the review part of their business.

They call it a 'Landscape Score' so I guess the logic is that you're shooting on a tripod using base ISO.

The point is that when you take a series of measurements, then aggregate them into a score with arbitrarily chosen weightings that aren't universally applicable, the scores are useless. 

Just doesn't follow. The score does not need to be "universally applicable" to be useful.

I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not. 
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 03:06:13 PM
Wow, I never knew this.  I figured they took several measurements 100, 400, 800, 1600, 3200, 6400 and THEN averaged them into a final DR number? 

No, they don't do this (and it would be a terrible idea).  They do take several measurements, but the published use case number comes from base ISO. Never a good idea to rely on guesswork to determine how they got the number.

Quote
I don't understand why you wouldn't do it as an average across the ISO range?    Especially when you consider that most photographers outside of a studio use the full ISO range.

The use case score is called "landscape" suggesting stationary subject matter that is typically shot at base ISO. Landscape photographers have tripods.

There is a high ISO use case score which measures how well image quality holds up when ISO is cranked up (it basically determines the highest ISO setting for which predetermined SNR and dynamic range criteria are met).

Quote
75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots

Title: Re: DxOMark vs. Reality
Post by: Zlatko on December 15, 2012, 03:23:55 PM
Just doesn't follow. The score does not need to be "universally applicable" to be useful. Indeed, the very notion of "usefulness" implies some degree of pragmatism which implies some amount of imperfection and/or compromise. The aggregate score is probably in fact quite useful for its target audience. Suppose you're a naive consumer looking for some kind of step up camera, and you are looking at the bewildering array of consumer SLRs, superzooms, high end point and shoots, and mirrorless cameras. The single score measurement does provide a pretty good indicator of which camera is capable of better image quality. Even the single score will pretty quickly tell the consumer that there is a size / image quality / zoom tradeoff.
I thought we were starting to agree that the Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?  Oh well.  If the writer on Luminous Landscape is correct, then the Overall Score is "biased toward single shot HDR at low ISO settings" — a capability we never had in the past and which we may infrequently need.  If the Overall Score has to have a bias, that doesn't even sound like the correct bias to have.  Likewise, the Portrait Score essentially measures "chroma noise in the dark parts of a low-ISO image" — not exactly the measure of quality when we choose a camera for portrait photography.  :)
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 03:30:16 PM
I'll acknowledge that there is probably no such thing as 'universally applicable'.  But if the arbitrary weightings in the score are not aligned to the user's need/preferences, they are useless to that individual.  If the forumla/weightings are not fully disclosed, it's impossible to judge if they're applicable for a particular individual.

That's true, but it's not the same thing as saying that there do not exist individuals for whom it's useful.  The user for whom such a blunt instrument is useful is not even going to go to the trouble of addressing whether or not the score addresses their use case(s).

It would be better if the aggregate scores were disclosed, though if they can be reverse engineered within a point or two I don't see this as a major issue (again consumers of this aggregate score would not pay attention to the formula if it were published anyway)

Quote
Say I shoot landscapes at night (higher ISO to avoid star trails), or am unable to bring a tripod?  How does DR at base ISO help me evaluate a sensor for my needs?  The measurement is useful, the Landscape Score is not.

I don't see how coming up with a benchmark for every conceivable use case (or just not publishing use case benchmarks at all, thus forcing everyone to go through the measurements)  would make things any more clear or less confusing.

So it's true that some use cases might require more attention to detail on part of the user. The good news is that DxO make it all available.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 15, 2012, 03:37:58 PM
I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)

Photographers "should" pay attention to the actual measurements and properly understand sensor benchmarking.

However, not all "buyers of cameras" are serious photographers, and not all of them are as attentive to the nuances of sensor benchmarking as photographers should be.

Reviewing the raw measurements is better than reviewing the use case scores, which is better than reviewing the aggregate score, which is better than not reviewing any sensor benchmark.
Title: Re: DxOMark vs. Reality
Post by: Zlatko on December 15, 2012, 03:59:00 PM
I thought we were starting to agree that the  Overall Score may not be very meaningful to a lot of photography and should be ignored by photographers?

Well, I'm not sure that we really agree, because my position is that you should look at DxO's measurements instead of the aggregate score. Your position is that you shouldn't look at the aggregate score, but you are silent on whether or not someone who cares about sensor performance should look at DxO's measurements (or, if they care about sensor performance but not DxO, what alternative they should look to instead)
I think photographers should look at the measurements when they measure something of relevance to the photographer, and ignore the Overall Score.
Title: Re: DxOMark vs. Reality
Post by: Kernuak on December 15, 2012, 04:00:39 PM

Quote
75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots
Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.
Title: Re: DxOMark vs. Reality
Post by: leolol on December 15, 2012, 05:02:13 PM
@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.
Title: Re: DxOMark vs. Reality
Post by: Kernuak on December 15, 2012, 05:08:26 PM

Quote
75% of what I shoot is ISO 800 and above, and I'm sure I'm not alone....therefore their score for DR is almost worthless to me.....AND misleading, since most of us are just going to look at the score, not the method/process for achieving the score.

It's called "landscape" and is well documented, so it's hardly misleading. If you're shooting at ISO 800 or higher, you should be looking either at the high ISO use case score or the measurement plots
Actually, I disagree with their high ISO scoring system and that is based on my experience of real world low light, high ISO shooting. They still set an arbitrary figure of 9 stops of DR on the high ISO scores and give it equal weighting to the amount of noise. However, when shooting in low light, the dynamic range is usually very low, which makes the weighting irrelevant. If they applied appropriate weighting to low noise, there would potentially be very different results. I'm sure the gap between the D800/800E and the D3s, D4, 1D X and 5D MkIII would be much higher in real world shooting in those conditions. Of course, I haven't measured it, so can't make a categorical assertion, but it is certainly my feeling, based on experience and also by looking at images online. The latter four cameras are designed to cope with low light conditions, so it is reasonable to expect that, that is the case. That is in fact my biggest gripe with DxO, that their separate scores are too arbitrary and don't appear to be based on any real world conditions. Perhaps they do say somewhere, but it would help if they published how they came to deciding on what factors to include in their scoring systems.

I try to follow you but do not understand what you mean.
please explain
In low light conditions, the dynamic range in a scene is much less than any modern sensor can capture, therefore whether or not one sensor can capture more DR than another is irrelevant. As I've never measured it, it is rather subjective, but I would estimate that DR is in the 6 stop range for many of the low light conditions that pushed me into buying the 5D MkIII (that's purely an example and wasn't the only reason for buying it). When you start getting into that sort of lighting, then a test should be looking more at the amount of noise in an image and less at how much DR the sensor can capture. DR is desirable for some photography, but unimportant in others and any test of a sensor should reflect that in any objective testing. What I would like to know, is how DxO come up with their cut-off points. For example, why 9 stops of DR for low light testing? Why not 10 or 8? Like Neuro and one or two others, I have been involved in scientific research (albeit nothing to do with optics) and I know that statistics can be used to emphasise pretty much anything if you don't look too closely. That's why it is important to do the right tests and the right analysis. You get it wrong and don't apply proper rigor to the testing in medical research and you could put patients' lives at risk further down the line.
Ultimately though, photography is an art form and therefore subjective. What one person might find acceptable, someone else might find it objectionable. Nowhere is this more obvious than noise and some even add noise to add atmosphere. Even blown highlights or blocked out shadows can add to an image if done correctly, although again, some people will hate it while others love it.
Title: Re: DxOMark vs. Reality
Post by: Kernuak on December 15, 2012, 05:11:43 PM
@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

Lets take the german/russian trabi: Cheap Car, but there where better ones. Nevertheless it sold. Was it the best? No.
Actually, I don't think he was saying any camera or sensor was better than another, rather that the perception of "better" (however it is measured) is actually irrelevant. The only thing that matters for a business is sales figures. If sales or profits are increasing, then they are achieving their goals (at least in most cases).
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 15, 2012, 05:41:55 PM
@Neuro
You are always basing your conclusions for "who makes better cameras" on sales. But thats not how it works, you cant judge a camera on its sales.

No, I base my conclusions on which camera is better (for me) using criteria and performance aspects that matter to me.  In my opinion, that's what everyone buying a dSLR should use to make their own decision, and in most cases, that's likely the case.

It is a fact that Canon sells more dSLRs than Nikon.

Takes together, that suggests that in aggregate, more people have judged that Canon dSLRs satisfy more of their decision-making criteria.

Consider - there's an election, at the end of which (hanging chads notwithstanding), there's a winning candidate and a losing candidate. 'Winner' is based on a count of votes.  You voted for the candidate you thought was 'the best'...but that candidate may not have won.  You're entitled to your opinion, and if you picked the losing candidate, that doesn't invalidate your opinion. But at the end of the counting, you need to understand that the candidate you liked best lost, and your opinion is in the minority.

There are lots of ways to judge something.  The problem is that the vast majority of them are biased by personal opinion.  What's best for you isn't necessarily best for everyone. I use sales figures as a metric because that's one of very few unbiased metrics available.  We could argue all day about which candidate would be 'best for the country' or which camera is 'the best for taking pictures'.  But there was an election, with a winner and a loser...and Canon has sold more cameras than Nikon.

Those are the facts, plain and simple.  If it helps, while you don't have the option to be governed/represented by the losing candidate, you can go buy a Nikon dSLR.
Title: Re: DxOMark vs. Reality
Post by: tortilla on December 16, 2012, 04:11:14 AM
A skilled photographer is not the same as  he or she  is a good  copyist and know how a digital work flou works.  Have you  anytime worked in a darkroom with a negative film  then you will understand what I mean and likewise it is with a raw files.

A skilled photographer with also skills in the digital "dark room" kan do so much more than a photographer with no  photoshops skills.  And even more with a camera who has 14 stops DR.
best regards , pro since 30 years back.

Well, when you need 14 stops DR to "rescue" your pictures via Photoshop, then you should ask yourself, if you`re a real pro.
When it comes to the the point where you really need 14 stops of DR, you've taken your picture in a wrong way.

And: none will see the difference. Neither a customer nor other pro photographer.

So, does that mean HDR imaging is only for amateurs like me who are not able to exposure right, because they don't find the full-automatic setting? ;)  Real world lighting situations can sometimes have way above 14 EV DR.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 16, 2012, 07:29:41 PM
I prefer to look at trends rather than specific points in time, i.e I'm more interested in knowing if Canon/Nikon's DSLR share is growing/shrinking than what specific percentage it is.

That's fine, as long as you realize that one quarter does not equal a trend...  FWIW, Canon has gained market share at Nikon's expense for the past 5 years or so. 
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 17, 2012, 06:25:13 AM
Well it is hard to see Canon maintaining its sales numbers without continued fire sales of (for example) the 5D3 at a discount of over $500 of the MSRP. Or one might say that such fire sales are evidence that vendors are struggling to sell 5D3 stock at MSRP and that Canon sales numbers will suffer if they don't happen.

Fire sale...hyperbolize much?  As for sales numbers 'suffering' I just had a look at Amazon's dSLR sales rankings this busy holiday shopping season, and the top three best sellers are from....Canon (top four on the list, but the Canon T4i is #1 and #4, with different lenses).  And while we all know the D800 is a much better camera (so telleth us the almighty DxOMark), and it's cheaper, too, the 5DIII is in the Top 20 (#14), and the D800 is not (#21).  Canon does seem to be suffering badly, oh my.  ::)
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 17, 2012, 09:59:57 AM
Is Amazon a reference?

It's a reference in the sense that one can see the data, so claims can be substantiated.  Do you have data that you can show to substantiate the claim than the D800 is outselling the 5DIII 20:1 in Sweden? 

I wonder how many cameras are sold in Sweden, compared to the USA?  There are ~33 people in the US for every 1 person in Sweden.  Raw numbers matter far more than ratios.

Nikon V1 best selling camera in Europe

I have read (in a Nikon press release) that the Nikon 1 was identified as the best selling compact camera system in Europe in 2012.  Where was it published that the Nikon V1 was the best selling camera in Europe?

Nikon has increased  theres SLR selling with 26% 2012.

Not exactly.  I think you're looking at the results for the 1st half of their FY2013, which are the 2nd and 3rd quarters of the 2012 calendar year (Canon and Nikon's fiscal years are offset).  Performance in the first calendar quarter of the year were less impressive, barely making up for the loss in the last half of 2011. 

Over those same two quarters (2Q12 and 3Q12), Canon reported a gain of 47% and a loss of 7%, respectively, in the dSLR segment.  Which is better, when looked at over a 6 month period - a 26% gain, or a 47% gain for the first 3 months and a 7% loss for the next 3 months?
Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 17, 2012, 10:40:15 AM
Here's some statistics from flickr.   I don't know how relevant this is but this shows the most used cameras by flickr users.

http://www.flickr.com/cameras/ (http://www.flickr.com/cameras/)
Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 17, 2012, 10:41:45 AM
Hi Mike, but sorry we can't see anything... Can you re-post the links of your sources?  Thanks.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 17, 2012, 11:27:42 AM
What I show you  is that there a world out side yours
I'm well aware of that, thanks.  I'm quoting sales figures from Canon and Nikon reflecting global sales, and aggregated market reports, and those show that Canon has a much larger market share than Nikon, that Canon has outsold Nikon year over year for the past 5 years, with the exception of 3Q12, and that means that Canon is the world leader in dSLR sales.  "Best and most beautiful" are subjective, billions of ¥ in revenue and number of units sold are not. 

Nikon SLR NR 1 selling in Japan...in  2011
Ok, so let's look at that.  There are two ways to interpret that statement, either Nikon was #1 in Japan in 2011, or Nikon had the #1 selling model in Japan in 2011.  The D3100 was the best selling dSLR in Japan in 2011.  The fact that the D3100 was the best selling dSLR in Japan in 2011 is misleading - the D3100 was the current model in it's segment for the entire year; in that segment, Canon's Kiss X4 was the current model at the beginning of the year, and was replaced by the Kiss X5 during 2011.  If you add the Kiss X4 and Kiss X5 sales together, that far exceeds D3100 sales.

Moreover, 6 of the top 20 cameras on that list (http://thedigitalvisual.com/the-nikon-d3100-was-the-best-selling-dslr-in-japan-for-2011) are Canon models compared to 4 Nikon models on the list, and if you add up the sales by vendor, Canon dSLRs had 27% of the Japan market in 2011, compared to 22.5% for Nikon.  So...was Nikon really #1 in Japan in 2011?  No.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 17, 2012, 02:38:32 PM
rather uninteresting who sells the most SLRs - right?

To shareholders, such things are very interesting...and because publicly-traded companies have an obligation to their shareholders (it's their primary obligation, in fact), such things are of paramount importance to those companies.  If it's important to them, it should be important to consumers of their products.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 17, 2012, 03:00:55 PM
So I guess DxO scores have a negative impact on CAJ share prices, and the dividend CAJ pays.
We should watch DxO's new scoring system for lenses closely, we may want to short the stock.
Title: Re: DxOMark vs. Reality
Post by: sdsr on December 17, 2012, 03:29:50 PM

Well it is hard to see Canon maintaining its sales numbers without continued fire sales of (for example) the 5D3 at a discount of over $500 of the MSRP. Or one might say that such fire sales are evidence that vendors are struggling to sell 5D3 stock at MSRP and that Canon sales numbers will suffer if they don't happen.

If "fire sales" matter, what would you have us infer from the current $700 discount on the D600 + kit lens via certain (all?) US (only US?) vendors?
Title: Re: DxOMark vs. Reality
Post by: weekendshooter on December 17, 2012, 03:48:17 PM

Well it is hard to see Canon maintaining its sales numbers without continued fire sales of (for example) the 5D3 at a discount of over $500 of the MSRP. Or one might say that such fire sales are evidence that vendors are struggling to sell 5D3 stock at MSRP and that Canon sales numbers will suffer if they don't happen.

If "fire sales" matter, what would you have us infer from the current $700 discount on the D600 + kit lens via certain (all?) US (only US?) vendors?

Probably that they have way too many of the terrible kit lens left over.
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 17, 2012, 04:20:56 PM
There is some nitpicking about the way those measurements are aggregated but that's about it.

Subaru Legacy, Overall Score = 92
BMW 760Li xDrive, Overall Score = 84

Preposterous?  Well...the Overall Score is based on a weighted composite of two Use Case Scores, Winter Utility and Summer Utility. Those are based, respectively, on accurate and reliable Measurements of the ability of just the left rear wheel to push the car up a 20-degree incline, and the towing capacity.  But those details are just nitpicking. The Overall Scores clearly show that the Subaru is better.

 ::)

Well, IMO it's actually worse than this.  Because they would publish overall scores, but those scores are only reflective of the engine performance.  There is no consideration for handling, traction, braking, interior features, etc.  I don't know anyone who would buy a car based on engine performance charts, yet somehow this makes sense in a camera? 

"My car has more ponies than yours and is therefore better even though I can't corner over 15mph, and it becomes airborn over 85mph."
Title: Re: DxOMark vs. Reality
Post by: marinien on December 17, 2012, 06:57:57 PM
http://www.techradar.com/news/photography-video-capture/cameras/nikon-d3100-tops-japans-best-seller-list-1051269 (http://www.techradar.com/news/photography-video-capture/cameras/nikon-d3100-tops-japans-best-seller-list-1051269)
said nothing else

http://www.bjp-online.com/british-journal-of-photography/news/2040291/nikon-adds-digital-slr-range-d5100 (http://www.bjp-online.com/british-journal-of-photography/news/2040291/nikon-adds-digital-slr-range-d5100)
said nothing else.

rather uninteresting who sells the most SLRs - right?
or , as someone else member told  you, Trabants sold most of all cars in the old East. Toyota is best selling cars today.
yard stick?
over and out regarding this subject

So Mikael, if someone posts an article solely about high ISO performance, your conclusion would be: DR at low ISO is "uninteresting", right? Right!

Your example with Toyota just doesn't work. Canon and Nikon are in the same market segments for DSRL, and they have , I dare to say, comparable distribution networks in most countries, so sale figure does mean something about user preference. While Toyota and, say, Mercedes are not in the same market segment, or Toyota and Peugeot do not have comparable global distribution networks, so their sale figures do not have the same meanings in user preference as Canon vs. Nikon.

Nikon SLR NR 1 selling in Japan...in  2011
Ok, so let's look at that.  There are two ways to interpret that statement, either Nikon was #1 in Japan in 2011, or Nikon had the #1 selling model in Japan in 2011.  The D3100 was the best selling dSLR in Japan in 2011.  The fact that the D3100 was the best selling dSLR in Japan in 2011 is misleading - the D3100 was the current model in it's segment for the entire year; in that segment, Canon's Kiss X4 was the current model at the beginning of the year, and was replaced by the Kiss X5 during 2011.  If you add the Kiss X4 and Kiss X5 sales together, that far exceeds D3100 sales.

Mikael, next time when you want to argue, try to learn from Neuro. Above is what I consider a good argument (throwing in some numbers would make it better) ;)

Last advice: how about deciding once and for all on having a space or not following a comma in your signature? It's a detail, but it does reveal something ...  ::)
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 17, 2012, 07:21:27 PM

Mikael, next time when you want to argue, try to learn from Neuro. Above is what I consider a good argument (throwing in some numbers would make it better) ;)

Last advice: how about deciding once and for all on having a space or not following a comma in your signature? It's a detail, but it does reveal something ...  ::)

What would that reveal?
Not real sure on this one?

Title: Re: DxOMark vs. Reality
Post by: marinien on December 17, 2012, 07:33:22 PM

Mikael, next time when you want to argue, try to learn from Neuro. Above is what I consider a good argument (throwing in some numbers would make it better) ;)

Last advice: how about deciding once and for all on having a space or not following a comma in your signature? It's a detail, but it does reveal something ...  ::)

What would that reveal?
Not real sure on this one?

I am a researcher in applied mathematics. I work with a lot of data and stats. Being rigorous in every step is a key element. A misinterpretation here, a bad format there and the results may mean nothing. Our Mikael posted and repeated again and again his tests on this forum. But by looking at his signature, I'd not have a great confidence in his conclusions  ;).
Title: Re: DxOMark vs. Reality
Post by: elflord on December 17, 2012, 07:35:52 PM
Well, IMO it's actually worse than this.  Because they would publish overall scores, but those scores are only reflective of the engine performance.  There is no consideration for handling, traction, braking, interior features, etc.

If they tried to incorporate all of those features into their score, it would make things worse (and make the score more open to criticism).

It's better that they stick with a single thing -- benchmarking sensor performance.

Quote
I don't know anyone who would buy a car based on engine performance charts, yet somehow this makes sense in a camera? 

This is a straw man. No one has proposed that sensor performance is the only factor a buyer should consider.

You seem to be suggesting that no-one should measure engine performance (and therefore no-one should publish engine power/torque  output and curves)
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 17, 2012, 07:38:39 PM

I am a researcher in applied mathematics. I work with a lot of data and stats. Being rigorous in every step is a key element. A misinterpretation here, a bad format there and the results may mean nothing. Our Mikael posted and repeated again and again his tests on this forum. But by looking at his signature, I'd not have a great confidence in his conclusions  ;).

I see,

This is why I do not trust doctors. Have you seen their signatures.
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 18, 2012, 06:25:14 PM
Well, IMO it's actually worse than this.  Because they would publish overall scores, but those scores are only reflective of the engine performance.  There is no consideration for handling, traction, braking, interior features, etc.

If they tried to incorporate all of those features into their score, it would make things worse (and make the score more open to criticism).

It's better that they stick with a single thing -- benchmarking sensor performance.

Quote
I don't know anyone who would buy a car based on engine performance charts, yet somehow this makes sense in a camera? 

This is a straw man. No one has proposed that sensor performance is the only factor a buyer should consider.

You seem to be suggesting that no-one should measure engine performance (and therefore no-one should publish engine power/torque  output and curves)

Not saying no one should measure, but that in the camera world these measures are blown way out of proportion for what they should be.  And if they're truly just measuring sensors, then shouldn't be the rating be identified to the sensor, and not the camera??  Pretty sure multiple camera's use the same sensor, but you don't find the sensor score, you find the camera score.

My gripe is not in their testing, it's in the presentation and marketing layer.  For example, they say "The Overall Sensor Score is based on all characteristics of a camera sensor, independent of the camera lens..."  Umm, it's independent of a whole lot more than just the lens.  My point is there is a certain amount of deceptiveness in their marketing (for lack of a better word.)  They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities.  Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria.  I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score.  Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 18, 2012, 07:01:27 PM
Not saying no one should measure, but that in the camera world these measures are blown way out of proportion for what they should be.  And if they're truly just measuring sensors, then shouldn't be the rating be identified to the sensor, and not the camera??  Pretty sure multiple camera's use the same sensor, but you don't find the sensor score, you find the camera score.

Under the "cameras" tab on their page, I see "camera sensor database", "camera sensor ratings" and "compare camera sensors".

I don't think they're making it a secret that they are benchmarking sensors.

Quote
They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities.  Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria.

What descriptions would be more appropriate in your opinion ?

Quote
I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score.  Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.

How precisely are they trying to "mislead" people ? This is a step beyond simply criticising their choice of naming, here you appear to be accusing them of choosing the names in such a way as to favour their business.

Well, please do substantiate this accusation a little. For example, how do they benefit from calling their high ISO use case "Sports" instead of "event photography" or "wedding photography" ?

Also, it's clear that you don't like their choice of names. It's not clear (a) what you would choose instead, or even (b) if your choices, with the benefit of a few years of hindsight (e.g. after everyone has had their chance to publically critique DxO's), are any better.

Title: Re: DxOMark vs. Reality
Post by: tnargs on December 18, 2012, 08:04:38 PM
This is a straw man. No one has proposed that sensor performance is the only factor a buyer should consider.

You are joking of course. The number of posts and comments on the internet basically saying "I have to jump ship from Canon to Nikon NOW, and so do you, because the Nikon (Sony) sensor has a higher DR" is fast approaching infinity.   8)
Title: Re: DxOMark vs. Reality
Post by: elflord on December 18, 2012, 09:43:08 PM
I do not pretend to be a technical expert in camera gear....BUT....it's interesting to notice that the nikon d600, WITH THE 2ND HIGHEST RATED SENSOR SCORE EVER BY DXO, (only behind the d800) WITH A SCORE OF 94, is now being heavily discounted by nikon, not by retailers, but by nikon....just months after it's launch.  (a couple months ago you could have had the camera for $2k, now you can get the camera and the nikon 24-85 ($600) for $2k.   

Yes, and these have  higher than the D4, which costs how much ?

People buy cameras, not sensors.

Quote
it's just amazing to me that people believe that somehow a $2000 nikon entry level full frame camera somehow has a sensor(according to the experts at dxo...lol)  that is SIGNIFICANTLY HIGHER than the $6800 canon flagship 1dx.

all around the world people understand this phenomena...it's called $$$$$ under the table.

So should DxO instead award higher sensor benchmark scores to cameras that cost more, so as not to offend snobs who own or worship expensive equipment ?

It's not surprising at all that the D600 gets a similar score to the D800. The sensor is the same size, same vintage, and made by the same manufacturer. Why would it not have similar performance characteristics ?
Title: Re: DxOMark vs. Reality
Post by: elflord on December 18, 2012, 10:12:15 PM
hey ELF...i already implied why....read again the last part of my post.

as I've already stated on this forum...it's of my opinion, that DXO is a group of lazy and unorganized....enough said.
I do understand that you're casting aspersions as opposed to stating your views in a more forthright manner (which might create an expectation that you substantiate your allegations).

What you do not understand is that I'm asking you to put up or shut up. If you have some insight or knowledge that supports your view, please do not be shy -- share it with us. If you don't have any such insight or knowledge, it does raise some questions as to why you are prone to make unsubstantiated and unprovoked attacks on the reputation and character of DxO.
Title: Re: DxOMark vs. Reality
Post by: Northstar on December 18, 2012, 11:13:43 PM
hey ELF...i already implied why....read again the last part of my post.

as I've already stated on this forum...it's of my opinion, that DXO is a group of lazy and unorganized....enough said.
I do understand that you're casting aspersions as opposed to stating your views in a more forthright manner (which might create an expectation that you substantiate your allegations).

What you do not understand is that I'm asking you to put up or shut up. If you have some insight or knowledge that supports your view, please do not be shy -- share it with us. If you don't have any such insight or knowledge, it does raise some questions as to why you are prone to make unsubstantiated and unprovoked attacks on the reputation and character of DxO.


look up my recent hisotory on posting about DXO...i just don't believe they have their "act" together.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 08:50:53 AM
What you do not understand is that I'm asking you to put up or shut up.

That has been asked of others on this thread, with no 'putting up' evident. 

you have the right to think what you want.
we are some who do not share your opinion, you must also accept that

+1  Without supporting evidence, statements are merely opinion, and as you say, we are all entitled to our own with no expectation that others will agree.

Just to remind people of the central thesis of this thread - it's not that DxOMark scores are 'cooked' or 'false', nor even that the scores are biased in favor one brand over another.  I've seen no evidence of the former (quite the opposite), and while the latter may be true, there's no evidence it's driven by anything other than logic - we generally try to shoot at the lowest ISO possible, so there's some logic to biasing the scores to base ISO. 

Rather, the central point is that DxOMark's scores seem to have had no meaningful impact on the real world aggregate buying decisions of consumers, as demonstrated by Canon's and Nikon's own financial reports detailing their global sales figures.  Put succinctly as a statement comprising fact, Nikon dSLRs have beaten Canon's in DxOMark scores for several years, and Canon's dSLRs have outsold Nikon's for those same several years. 
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 19, 2012, 08:58:33 AM
make unsubstantiated and unprovoked attacks on the reputation and character of DxO.

Interesting statement;

I think anytime a company presents information to the public, if it is misleading in nature it calls in to question the character of the company.

The tests DXO performs on sensors, if the data is presented for what it is I would not question their presentation or motive. When they take the data and manipulate it to arrive at a score they created with a description that they perceive or make us want to believe is correct, I have to question why they would do this.

How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.
Title: Re: DxOMark vs. Reality
Post by: Maui5150 on December 19, 2012, 09:23:24 AM
DxOMark scores are junk and meaningless.  While there may be a "methodology" to their testing, their score are meaningless for comparison between two cameras, and this is not a Nikon versus Canon debate, but when you see clearly more advanced and better performing camera producing lower results, that alone is justification that the significance of their numbers is in fact, insignificant. 

Is the D600 5 points better than the D4? 

According to dxOMarks, the D600 is a 94 and one of the best cameras tested, and the D4 a lousy 89. 

By DxOMarks scores, the Nikon flagship is crap and over priced.

As well... how can you have a score of "94" and then try shooting above ISO 3200 and see what you get for results. 

If money was not an object, would you buy a D4 or a D600?    According to Dx0marks you are an idiot if you buy the D4 because the D600 is vastly a better scoring camera...

Then again... I think the real idiots are the one who pay attention to Dx0Marks.

Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 19, 2012, 10:20:58 AM
So what's your point Mikael?  Did we learn anything new again?  We already know those figures and we've seen it already.  So with those figures, you want to drill us that we should switch to Nikon because it's better? Here we go again...   :)  Reality is, there's more to a camera than a sensor.  Just like there's more to a woman than a pretty face.  If we choose Canon, does that mean it is wrong?  It has been long ago since we had accepted that Sony's sensors are better than Canon but what's the point?  Most of us made our decision because we considered the camera as a whole and we find Canon more attractive than Nikon.  Hmmm... Are you sure you're not a marketing agent of Nikon?  :D
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 10:30:49 AM
DxOMark scores are junk and meaningless.
Please point why and  how the DXO sensor measurement is faulty.
Note that Maui150 stated 'scores', not 'measurements'.  IMO, their sensor Measurements are fine, it's their Sensor Scores, that are faulty. 

First, those scores are derived from the measurements in an incompletely disclosed manner - it's a 'weighted average' but weighted how? DxO themselves have stated, "The DxOMark Camera Sensor score is under normal conditions a weighted average of noise, dynamic range and color sensitivity information. But some non-linearities are deliberately included in the algorithm to avoid a clear weakness in one area from being masked by a strength in other areas," (source (http://www.luminous-landscape.com/essays/dxomark_sensor_for_benchmarking_cameras2.shtml)).  An analogy might be the Dow Jones Industrial Average, which is a price-weighted index - what if the Dow decided to give five of those 30 companies greater weight in the index, but didn't tell us which companies were the chosen five, or whether it was the same five from day to day?  If they did that, the DJIA would have little utility as an index, although we'd still know the closing share prices of the 30 index companies, much like the DxOMark Scores have little utility as a sensor benchmark, although we know the results of the individual DxOMark measurements.

Second, the Overall Score is biased toward performance at base ISO.  Two of the three subscores (Landscape and Portrait) consider performance only at base ISO, rather than considering performance of those metrics across the range of available ISOs for the sensor being tested.  Not all amplifiers are created equal, and DxO's measurements clearly show that when comparing two sensors, while one may have better DR and color depth at base ISO (e.g. 100), the other may have better DR and color depth at ISO 3200.  However, only base ISO contributes to those subscores.  That's a bias in the subscores, and thus, it's carried forward into the overall score.

Now...put those two together.  An Overall Score derived from three subscores with two considering only base ISO, meaning a 2:1 bias in favor of base ISO performance.  An Overall Score which is a weighted average of three subscores with unknown weightings, affected by intentionally-selected but nondisclosed nonlinearities.  Do the weightings and undisclosed skewings of the algorithm correct for that potential 2:1 bias, or make it worse?  We don't know. 

So, while it's possible to look at the specific measurements in their data (and I applaud DxO for publishing those data), and while it's possible to post those measurement data over and over again, it doesn't change the fact that DxOMark's Overall Score and Use-Case Scores are derived from those data in an ambiguous and undisclosed manner, and that makes them faulty.   
Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 19, 2012, 10:39:43 AM
My point is, why troing garbage at DXO? noting else

I respect the scores of DXO but I take them with a grain of salt.  As Neuro pointed out, if you (DXO) really want to give measurements/scores you need to divulge everything and take everything into account (no bias, take equally everything into consideration).  Without doing that, you'll never be able to escape scrutiny.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 19, 2012, 11:03:37 AM
DXO seems for all intents to do exactly what they claim to do: take precise, repeatable measurements of certain aspects of digital camera sensors, and condense these measurements into continually more and more abstract performance figures that are easier to read and remember but harder to relate to physics. This is similar to how a camera review site might give a camera a score of 67 out of 100 points - except that in that case the reviewers subjective opinion tends to be a factor.


I think the point is the "abstract performance figures".
I think you hit the point, a camera review site may gvie a score, but when DxO decides how to create their "abstract performance figures" they inject their subjective opinion in to how the "abstract performance figures" are presented. This is the flaw in their method. They are doing the same thing a camera  (sensor)review site would do.

Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 11:10:56 AM
Peter: So how fair is the DxOMark Camera Sensor Score?
There is a difficult but important question. Probably every image quality scientist[74] in the world would have a somewhat different personal preference for a benchmark like this. But my impression is that the benchmark is pretty useful: I analyzed the model and the data, but didn’t find any serious flaws.

...then he goes on to discuss the low ISO bias which I mentioned, and he argues that DxOMark should release their methods for determining the overall score, the two points which I find most objectionable.  He also lists several other issues with DxOMark's process.

As he states, "If you compare the DxOMark data in Figure 7 for a number of prominent cameras you would get a more balanced impression about which camera to buy than by just looking at the overall DxOMark Sensor score. If you focus on the latter, you would strongly prefer the Nikon D800 with its excellent low ISO dynamic range. But this emphasizes one aspect of the sensor (essentially the ability to do single shot HDR) that provides a capability we never had in the past. It is a feature which we may infrequently need – and one that some types of users may never see (e.g. if you shoot JPG). However, at sufficiently high ISO, other models win. High ISO usage may be a more relevant usage for many users than HDR ability at low ISO."

That illustrates the issue with using DxOMark's Scores as a benchmark for sensors, and emphasizes the importance of examining the actual measurements (sort of like reading just the abstract of a scientific publication - you are relying on the author's conclusions drawn from their own data interpreted through the lens of their own biases, when it's preferable to examine the data for yourself and draw your own conclusions...which may or may not agree with those of the author).
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 11:30:23 AM
well , for me there are no problems, Im looking  for example at FWC, QE, read out noise , and then get  an idea of the sensors behavior, and then I look at the practical difference, DR, noise, banding etc by comparing the cameras and under the exact same parameters.
Which I have done 100 of times

Exactly.  But you are clearly approaching the issue of sensor performance with a much broader and deeper knowledge base than a typical consumer.  I would bet that most people who look at DxOMark comparisons have no idea what the abbreviations FWC and QE even stand for, much less what those concepts actually mean in terms of sensor performance.  Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions. 
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 19, 2012, 11:41:41 AM
Under the "cameras" tab on their page, I see "camera sensor database", "camera sensor ratings" and "compare camera sensors".

I don't think they're making it a secret that they are benchmarking sensors.

Not secret, just enough under the radar that it's easily overlooked by average consumers.

Quote
Quote
They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities.  Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria.

What descriptions would be more appropriate in your opinion ?

Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

Quote
Quote
I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score.  Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.

How precisely are they trying to "mislead" people ? This is a step beyond simply criticising their choice of naming, here you appear to be accusing them of choosing the names in such a way as to favour their business.

Well, please do substantiate this accusation a little. For example, how do they benefit from calling their high ISO use case "Sports" instead of "event photography" or "wedding photography" ?

I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon.  I don't know this for a fact or why they would do such; maybe they own stock.  But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports.  Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)
Title: Re: DxOMark vs. Reality
Post by: MarkII on December 19, 2012, 12:52:47 PM
Worse, DxOMark's Scores are recapitulated at review/comparison sites like snapsort.com, and used in overall camera scores, without reference to the actual measurements so consumers can make informed decisions.
To be fair, that is not really DXO's problem. The internet is stuffed full of sites that either accidentally or deliberately pick and choose numbers that give a biased impression.  Snapsort offends in many more grievous ways than their use of sensor scores.

And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions. There is so much information to digest, they would never get to the point of consuming anything. Though they do seem to get upset if anything suggests that their decision was less than perfect...

Maybe DXO should add thumbs-up and thumbs-down buttons. Click the button to show what you think about the score. By storing the information in a browser cookie, they could adjust their scoring metrics to suit each individiual taste :-)
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 12:58:09 PM
And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people.  But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 19, 2012, 02:09:56 PM
And frankly, I don't think that non-obsesed consumers care about making truly personally informed decisions.

Not sure I agree with that...I think most consumers make personally informed decisions - it's just that since it's personally informed, and what constitutes adequate information for that is different for different people.  But that idea is consistent with what I'm saying - DOMark data are not a factor in the buying decisions of most consumers when it omes to a dSLR purchase.

Ah, but DXOMark doesn't just do dSLR's.   ;)
Title: Re: DxOMark vs. Reality
Post by: Maui5150 on December 19, 2012, 03:02:23 PM

your answer is  clever like  as -  all Canons gear is worthless
Please point why and  how the DXO sensor measurement is faulty.

Which is a better overall camera?  The D4 or the D600?

Which will take the best picture?  The D4 or the D600?

Which has a better Dx0Mark Score? 

The answer to the first 2 is the D4, and any HONEST person would agree

Yet the Dx0 Mark for the D600 is over 5% higher? 

Sort of PROVES my point the the Dx0Mark score is MEANINGLESS

Or perhaps you are trying to say that Nikon throws poor sensors in their flagship camera (meaning that the D4 is SUPER over priced)
Title: Re: DxOMark vs. Reality
Post by: elflord on December 19, 2012, 06:55:21 PM
look up my recent hisotory on posting about DXO...i just don't believe they have their "act" together.

I did. It looks like you've gone on a fishing expedition to find some kind of perceived wrongdoing.  I suppose this is consistent with casting aspersions and not being forthright about what your true objections are. One recent post was a complaint about typos, the other about their lens benchmarking. None of this has any relevance to or casts any doubt on their status as the industry leader in sensor benchmarking.

It seems to me that you didn't like it that Canon cameras did not score well on their benchmarks and decided to smear them on that basis.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 19, 2012, 06:58:57 PM
Umm, how about "Color depth", "Dynamic Range", and "Low-light ISO"?

But they use those labels as well.

Quote
I guess I am just of the mind that their scores (not their measurements) seem to have a bent towards Nikon.  I don't know this for a fact or why they would do such; maybe they own stock.  But the net result (pun intended) is that an "I'm looking for a good sports camera" type of query returns values that are actually only good "low-light ISO" cameras and completely suck for sports.  Why would they perpetrate this unless they are either lazy (as one person suggested), have a hidden agenda, or just kinda clueless (which I doubt, because they seem pretty bright.)

Well, they don't benchmark AF systems.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 19, 2012, 07:03:33 PM
How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 19, 2012, 07:15:33 PM

your answer is  clever like  as -  all Canons gear is worthless
Please point why and  how the DXO sensor measurement is faulty.

Which is a better overall camera?  The D4 or the D600?

Which will take the best picture?  The D4 or the D600?

DxO do not attempt to measure which is "a better overall camera" (whatever that means though I think most would agree that it's the D4) nor do they measure "which will take the best picture" (what does "the best picture" mean ?)

They benchmark sensors.

Now, your answer to the question is ?
Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 19, 2012, 08:04:42 PM
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 08:21:29 PM
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores (http://www.dxomark.com/index.php/About/Sensor-scores)

Yep, reality is the whole point.  If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs.  Has that been happening?  Or does Canon still have a greater market share?
Title: Re: DxOMark vs. Reality
Post by: verysimplejason on December 19, 2012, 08:32:25 PM
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores (http://www.dxomark.com/index.php/About/Sensor-scores)

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance:  nada more.

DXO scores sensors while in REALITY there's more to a camera than a sensor.  Is it that hard to understand?  That's the problem with DXO and what's almost everybody is pointing out.  DXO seems content on grading the sensor but not the whole camera body.  That's why this is titled DXOMARK vs REALITY.
Title: Re: DxOMark vs. Reality
Post by: TheSuede on December 19, 2012, 08:44:35 PM
The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

As stated so many times before, you cannot put a "score" on a camera as a complete system for any sort of user average, you have to specify a quite tight definition of the score scope - what it does include and what it doesn't. And it does get tricky when you try to include more than one parameter in a "score".

You may (in my opinion...) for instance put a reasonably real-world relevant score on "low light noise performance". This score will then preferably accurately represent how much noise a camera will show in a shot given an average exposure of exactly "X" lumen seconds per mm2. The measurement "noise per lm*s/mm2" is just a strict definition of what a photographer would call "I want 1/200s at F2.8 in this light".

-But it (the score) will not say anything at all about how well the camera can AF in that same situation. Neither will it say anything about the viewfinder, the fps rate or the color accuracy. The score will be "low light NOISE performance", not "low light performance".
...

But the individual measurement scales and results in themselves that DxO use to build their total "score" are almost beyond reproach. You could possibly fault their presentation mode in some of the results, but not the results in themselves.
The few individuals and organizations that can replicate the accuracy and scientific stringency of their tests all arrive at the same conclusion - there's not much to be said about it except for the fact that DxO are almost 100% repeatable and accurate. The numbers just fit, and anyone measuring the same thing on the same cameras will arrive at the same result. And anyone comparing two cameras out in the real world can - if they adhere to reality in stead of maker-defined bogus ISO values and so on - also visualize exactly what that number means in an image.

So it's up to the reader of the results to make use of the information in a way that he or she can relate to, and here's where most people fall flat on their faces. Unfortunately DxO trip themselves up by trying to apply an overall "score" on a camera, but as far as anyone in the publishing business can tell - you NEED a promise of an end score or other simple graphical or single-digit grading system to get 95% of the readers to even bother starting to read.

If the average readers realizes that they actually might have to think a bit for themselves to get something out of something, they quite often just bugger off somewhere else before even starting to assimilate any real information.
Title: Re: DxOMark vs. Reality
Post by: TheSuede on December 19, 2012, 08:53:38 PM
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

kind of like saying
:"Everything Apple makes right now has to be crap, they're constantly dropping in market share"
???

Neither of those arguments are worth paying any attention to, since there's more to sales and market shares than measurable product quality.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 19, 2012, 09:12:48 PM
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

The DxO "scores" are in no way to be mistaken for "reality". They are what they are, an end sum of an arbitrarily weighed set of measurement points that in their constitution may (or may not!) in some way be representative of how you as an individual use your camera.

I could not agree more.  This was what I was driving at, initially.  At the time I originated this thread, there was a plethora of posts/threads proclaiming the superiority of the D800 (and other Nikon bodies) as cameras because of the DxOMark scores, and nearly as many posts/threads suggesting that Canon was doomed as a corporate entity because of their subpar sensors.  The former claim is clearly unsupported by DxOMark, and the latter claim is equally untrue.

Thanks, TheSuede, for your cogent post.   :)
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 19, 2012, 10:10:04 PM
If  we  shall discuss a specific topic (DXO measurements vs reality) let's us do that without involving other parameters.
it is very difficult to follow  when people are involving   other features in the cameras who has nothing to do with DXO measurements.

But REALITY does involve other parameters... Don't tell me you are taking pictures with just your sensor?  That is why this is titled DXO vs Reality.  Reality is what it is.  You need to consider everything and not only the sensor.  It's elementary my dear Watson...

Nope, DXO mark vs reality, you can read  at DXO what the marks includes  and which  sensor parameters  are measured. http://www.dxomark.com/index.php/About/Sensor-scores (http://www.dxomark.com/index.php/About/Sensor-scores)

To help photographers rank and compare photographic equipment, DxOMark provides four scores showing camera sensor image quality performance:  nada more.

The reality is that low-light ISO does not = a good sports camera, but that is how it's being portrayed.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 19, 2012, 10:14:53 PM
Yep, reality is the whole point.  If DxOMark = Reality, and their Scores were all that mattered, no one would be buying Canon dSLRs, only Nikon dSLRs.  Has that been happening?  Or does Canon still have a greater market share?

It is not as the thread title suggests, a "competitor" or an "alternative" to reality but rather a subset thereof.

BTW, if the scores do not substantially affect sales, why all the concern about them "misleading" the "consumer" ?
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 19, 2012, 11:41:15 PM
How I see DXO is they are a company that manipulates their findings to arrive at an arbitrary score they established, this is their reputation as I see it, and I hear it from others as well. You may have a different view of their reputation as many do. Again when a company starts receiving a bad reputation it calls in to question their character.

How are they fudging the data ? Do you have any evidence to support such "manipulation" besides hearsay ?

As to their reputation -- there are some who for whatever reason have a very strong affinity for certain camera brands. Let's call them "fans". All major brands have "fans". The fans due to their enthusiasm are often on internet forums. DxO does have a bad reputation among "fans" of brands that get low scores on their benchmark. This says more about said "fans" than anything else. They are overwhelmingly seen as the leader in sensor benchmarking.

Even the DxO sceptics here acknowledge and take seriously their measurements -- most of the complaints are with the methods used to aggregate those measurements.

Fudging? It wasn't my word.

I have all the evidence I need to may an informed decision. Thing is, I figured this out for myself several years back before I read a single thread about how bad DxO scoring is. I did it by reading how they do the testing and how they arrive at there scores. DxO provided this information to determine the flaw to their madness. It was easy with just a little intelligent reasoning to realize the overall scores are flawed. Did I need to consult someone other than DxO to make this determination?

Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?
Title: Re: DxOMark vs. Reality
Post by: elflord on December 20, 2012, 06:09:19 AM
Do you not read the posts you are responding to or do you just go in making accusations?
I am not sure, I say there method of testing is ok but the method they use to "aggregate those measurements" (your words) is flawed. And you go off on a different tangent.

"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.

Quote
Why do you feel the need to defend DxO. Did you buy your Nikon based on these scores and are afraid they may be make believe now?

I don't own a Nikon. I own a 5D Mk II and a Panasonic GF2. Before that I owned a Rebel. I am not a "fan" of either Canon, Panasonic or any other brand.

I defend them because in my opinion they do an excellent job at what they do, and most of the attacks are not well founded (the vast majority of criticism comes from camera "fans" who are unhappy that their manufacturer doesn't do well on the test scores. That and I don't have a very high opinion of camera "fans".
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 09:16:02 AM
"Manipulating the data" with the intent to mislead sounds like fudging to me, though it wasn't your word. As far as I can tell they are simply summarizing / aggregating the data and it just happens that with the current sensors, the way they do this doesn't work out well for Canon.


It has nothing to do with the Nikon/Canon debate as far as I am concerned.

You can read at DxO which numbers and factors they use. There are factors they leave out of the evaluation as well and they tell you this. The weight they apply to the three factors they use to arrive at the number score, is arrived at and determined by DxO. Why should we believe that the way they arrive at this average score is correct for how a sensor should be evaulated and scored? Is it because they "claim" to be the leader in sensor testing?

When I first went to DxO to use the information they provide to compare Canon cameras it was years ago and, Nikon only had few in the top ten. At that time when I looked at the scores they had just on Canon cameras the scores just didn't line up with reality with Canon vs Canon. If the final results do not line up with the real world situation, perhaps they should look at how they combine all this data to arrive at a final number score.

Then there is the description "landscape" or "sports" they use, rather than calling the test what it is actually for. Why don't they describe the test for what it is, and then imply  this is important for that activity. With landscaping DR is not the only thing that matters, why apply the name landscape to high DR?

I find DxO's presentation of the data to miss leading. Any time a large company chooses to put out data that on the surface appears to miss lead the general public I think you have to question their motives.

I do not believe the "simply summarizing" view. But then maybe in their minds the believe they are.
Title: Re: DxOMark vs. Reality
Post by: tnargs on December 20, 2012, 06:39:38 PM
I do not believe the "simply summarizing" view. But then maybe in their minds they believe they are.

Have you seen their new lens scoring thingy?   ::)

Clearly they haven't learnt their lesson. Blinkers?  8)
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 06:58:33 PM
I will se that day  Canon produces a camera with as good or better sensor  measurements than the Sony etc , how will you detractors of DXO look at DXO  measurements then??


The exact same way I have for years. Even before Nikon released all of the recent cameras that scored so high and Canon was on the top. Their method was wrong in the past, it is now and unless they change it will be wrong in the future no matter who's sensor scores a better make believe score.

Will you switch your view of DxO when Canon is on top of the chart?

Title: Re: DxOMark vs. Reality
Post by: elflord on December 20, 2012, 07:03:55 PM
You can read at DxO which numbers and factors they use. There are factors they leave out of the evaluation as well and they tell you this. The weight they apply to the three factors they use to arrive at the number score, is arrived at and determined by DxO. Why should we believe that the way they arrive at this average score is correct for how a sensor should be evaulated and scored? Is it because they "claim" to be the leader in sensor testing?

The way they choose to weight those factors is subjective. Therefore it is neither correct, nor incorrect. If you demand hard numbers as opposed to the wishy-washy subjective stuff, you will insist on seeing the measurements. I would feel very differently about DxO if they only showed us their summary numbers and not the underlying measurements.

As far as I'm concerned, they are welcome to publish whatever subjective summaries they like, though I don't care for these myself. Ultimately these I see as just another opinion -- well informed of course, but still just an opinion. There are other smart people who will have different opinions. The only opinion I care much for is my own after seeing the data.
 
Quote
When I first went to DxO to use the information they provide to compare Canon cameras it was years ago and, Nikon only had few in the top ten.

So I don't understand how they are trying to be "misleading" or what their alleged purpose is.

Quote
Then there is the description "landscape" or "sports" they use, rather than calling the test what it is actually for. Why don't they describe the test for what it is, and then imply  this is important for that activity. With landscaping DR is not the only thing that matters, why apply the name landscape to high DR?

I think the full name of the tests are (the parenthetical remarks are part of the name)

portrait (color depth),
landscape (dynamic range),
ports (low-light ISO)


I don't see what the problem is -- those who know what the technical terms mean, you can simply look past the non-technical ones.  For those who don't, omitting the non-technical terms doesn't help a whole lot.

Quote
I find DxO's presentation of the data to miss leading. Any time a large company chooses to put out data that on the surface appears to miss lead the general public I think you have to question their motives.

I still don't get who they are misleading and what they are trying to get their "victims" to do. So far we don't have a plausible motive for this alleged deception. According to the originator of this thread, no-one is fooled anyway, so not only we don't have a cause, we don't see an effect either. All we have is some grumbling from disgruntled camera fans, and additionally the fact that you take issue with some perceived inaccuracies in the way they summarize their data.
Title: Re: DxOMark vs. Reality
Post by: elflord on December 20, 2012, 07:05:47 PM
Will you switch your view of DxO when Canon is on top of the chart?

No, but I might upgrade my 5DII.
Title: Re: DxOMark vs. Reality
Post by: TheSuede on December 20, 2012, 07:44:43 PM
Have you seen their new lens scoring thingy?   ::)

Clearly they haven't learnt their lesson. Blinkers?  8)

I very much do NOT like their new lens score "thingy", and I especially dislike the fact that they removed the underlying base data from public view. But I know how to read MTF / image height and MTF / frequency diagrams - something I (without meaning to sound condescending) honestly believe very few average camera buyers do. I understand why they did it, but I don't like it.
Title: Re: DxOMark vs. Reality
Post by: TheSuede on December 20, 2012, 07:57:26 PM
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

(some stuff of the later part of the quote removed)

Actually (actually x3 now! :) ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 08:31:28 PM
I will se that day  Canon produces a camera with as good or better sensor  measurements than the Sony etc , how will you detractors of DXO look at DXO  measurements then??


The exact same way I have for years. Even before Nikon released all of the recent cameras that scored so high and Canon was on the top. Their method was wrong in the past, it is now and unless they change it will be wrong in the future no matter who's sensor scores a better make believe score.

Will you switch your view of DxO when Canon is on top of the chart?

your answers and statements are not worthy to be treated with a answer
there are plenty  answers already.

Why not? I answered your worthless question with a legitimate response.
Title: Re: DxOMark vs. Reality
Post by: Hobby Shooter on December 20, 2012, 08:32:26 PM
Isn't trying to use the argument:
-"They're wrong because Canon actually sell quite a lot of cameras"

Actually, the argument is, "They're irrelevant (or at least, not very relevant) because Canon sells quite a lot more cameras than their competitors who score much higher in DxOMark's rankings."

(some stuff of the later part of the quote removed)

Actually (actually x3 now! :) ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.
I would argue that irrelevant can not be graded in %, it's either relevant or irrelevant, binary. On the other hand something can be more or less relevant, but something can not be more or less irrelevant.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 20, 2012, 08:33:59 PM
Actually (actually x3 now! :) ) your argument needs to be refined even more... To reduce ambiguity to a level where the sentence actually makes any sense, it needs to be:
-"DxO scores in themselves are almost 100% irrelevant to camera market shares"

And that does not in any way mean that they are irrelevant to camera image quality. Which is what most people writing with a hostile intent towards DxO want it to mean.

Perhaps one more refinement: irrelevant to camera sensor image quality.  A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 08:39:26 PM

I still don't get who they are misleading and what they are trying to get their "victims" to do. So far we don't have a plausible motive for this alleged deception. According to the originator of this thread, no-one is fooled anyway, so not only we don't have a cause, we don't see an effect either. All we have is some grumbling from disgruntled camera fans, and additionally the fact that you take issue with some perceived inaccuracies in the way they summarize their data.

Not everyone researches as in depth as those on Canon Rumors. I know many with DSLR's that wouldn't have a clue how to interpret DxO's information, and they would just take the overall number at face value. What DxO's motive is I wouldn't know, but when ever a public company offers information there is usually a motive and it almost always involves monetary gain. Why do they post this information at all? Really I don't care.
Title: Re: DxOMark vs. Reality
Post by: Duprant on December 20, 2012, 09:07:30 PM
Not everyone researches as in depth as those on Canon Rumors. I know many with DSLR's that wouldn't have a clue how to interpret DxO's information, and they would just take the overall number at face value. What DxO's motive is I wouldn't know, but when ever a public company offers information there is usually a motive and it almost always involves monetary gain. Why do they post this information at all? Really I don't care.

Their motive is to publicize the DxO brand and sell DxO software. Nothing nefarious here.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 09:58:09 PM

Their motive is to publicize the DxO brand and sell DxO software. Nothing nefarious here.

What?? You don't think Nikon sent a can of Christmas popcorn or a care package over to DxO this year for all of their help?

Title: Re: DxOMark vs. Reality
Post by: elflord on December 20, 2012, 10:28:06 PM
Perhaps one more refinement: irrelevant to camera sensor image quality.  A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Well if your point is that the final product depends on aggregate performance of the whole system, that's fine, but what are they to do about it ?

They could aggregate AF performance into an even more subjective and debatable score, but you have repeatedly said that this is precisely what they shouldn't be doing.

Are you simply opposed to them publishing any sensor benchmarks because they expose the technological stagnation of your favourite camera brand ?
Title: Re: DxOMark vs. Reality
Post by: TheSuede on December 20, 2012, 10:29:03 PM
Perhaps one more refinement: irrelevant to camera sensor image quality.  A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Hehe... Yup - true.

But when you can 'assume with reasonable confidence' that two cameras have similar AF accuracy and sensitivity, the static image quality metric still rules.
-And you might have noticed that I deliberately MENTIONED the autofocus as being left out of the metric in the first place - so no modification is needed.

Better image quality = better image quality.
No modifier, no disclaimer, no BS sidetracking.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 20, 2012, 10:39:19 PM
Perhaps one more refinement: irrelevant to camera sensor image quality.  A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Well if your point is that the final product depends on aggregate performance of the whole system, that's fine, but what are they to do about it ?

They could aggregate AF performance into an even more subjective and debatable score, but you have repeatedly said that this is precisely what they shouldn't be doing.

Are you simply opposed to them publishing any sensor benchmarks because they expose the technological stagnation of your favourite camera brand ?

Sorry, but my reply wasn't directed generally, nor to you specifically. If you read my previous posts, hopefully you'd realize your questions are tangential and I'm saying none of the statements you seem to be expressing on my behalf.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on December 20, 2012, 11:01:19 PM
Perhaps one more refinement: irrelevant to camera sensor image quality.  A camera with a great sensor and lousy AF is likely to deliver poor image quality in many situations.

Hehe... Yup - true.

But when you can 'assume with reasonable confidence' that two cameras have similar AF accuracy and sensitivity, the static image quality metric still rules.
-And you might have noticed that I deliberately MENTIONED the autofocus as being left out of the metric in the first place - so no modification is needed.

Better image quality = better image quality.
No modifier, no disclaimer, no BS sidetracking.

Well, perhaps a modifier or disclaimer is in order.   ;)   For example, how about better image quality at ISO 12800?  That may be important to some, but not others, and it's not represented in DxOMark's Scores at all.  But that's why I've stated several times that I applaud them for making the measurement data available.  While I feel it's improper to publish Scores without disclosing full details of how those Scores are determined, personally, it doesn't matter to me for my own buying decisions.
Title: Re: DxOMark vs. Reality
Post by: PackLight on December 20, 2012, 11:06:03 PM
it doesn't matter to me for my own buying decisions.

Obviously, or you would have bought Nikon  :o
Title: Re: DxOMark vs. Reality
Post by: elflord on December 21, 2012, 06:06:29 AM
Sorry, but my reply wasn't directed generally, nor to you specifically. If you read my previous posts, hopefully you'd realize your questions are tangential and I'm saying none of the statements you seem to be expressing on my behalf.

After rereading I think I see what you were getting at -- it was just very puzzling on a first reading (and yes I understand that taken literally  it was inconsistent with what you've posted before which is why I found it puzzling)
Title: Re: DxOMark vs. Reality
Post by: bdunbar79 on December 21, 2012, 10:36:30 AM
Well it's like me and basketball.  In HS I couldn't miss in practice from the 3 point arc.  So you could say I was a great 3-point shooter.  However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open.  So it was actually meaningless.

We here on the forum know that this is just a sensor score.  Howver, consumers don't know that and if they just read the scores they automagically assume it is a better camera.  That's the problem.  It isn't.  Sensor is just ONE measured aspect of a camera and as a scientist, when I read their crap, it comes off to me as rather unscientific.  Whether it is or isn't, it just comes off that way to me.

You can defend it all you want, but it looks like crap to me and I don't use their scores at all.  Each to their own.
Title: Re: DxOMark vs. Reality
Post by: RLPhoto on December 21, 2012, 11:41:43 AM
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
Title: Re: DxOMark vs. Reality
Post by: thepancakeman on December 21, 2012, 11:52:28 AM
Well it's like me and basketball.  In HS I couldn't miss in practice from the 3 point arc.  So you could say I was a great 3-point shooter.  However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open.  So it was actually meaningless.

You, too, huh?  Excellent analogy!
Title: Re: DxOMark vs. Reality
Post by: bdunbar79 on December 21, 2012, 12:32:02 PM
Yeah then I had to "find another sport" a year or two later :).
Title: Re: DxOMark vs. Reality
Post by: tnargs on January 14, 2013, 10:22:55 PM
260 posts, how about a summary?  :)

Well, it seems that when people legitimately criticize DxO Labs' application of their measurements, apologists rush in to defend their measurements.

That's called an air ball, guys.

And when people point out that these DxO scores (applications) are being misused by review sites etc in a way that gives the wrong impression, apologists rush in to say that's not really DxO Labs' fault.

You must be joking.

Let us say Mercedes Inc engaged you to independently measure the quality of their product vis a vis say, Ford Inc, which they had also measured and knew would favour them. They also told you to summarize your findings and publish them. You go ahead and do it but goodness gracious, what's this? Your website is technically correct and shows Mercedes' quality advantage at the deepest level, but your way of summarizing and organizing and communicating gives the general impression that Ford is the better overall vehicle!

Suddenly, due to your excellent technical reputation, the world takes notice of your findings. Reviewers start to refer to 'The Ford Advantage' on their websites. Fleet buyers start recommending to boardrooms that current Mercedes contracts for trucks and buses should not be renewed, partly due to a new quality value matrix by an industry technical expert that favours other marques. The boardroom executive gossip soon reaches the ears of Mercedes executives.

Suddenly your client is on the phone to you in an absolute Teutonic fury. Yes, they say, your base measurements are correct but what on earth have you done with it? Why the blazes that way? Don't you realize how everyone is interpreting it?

Here is your answer: (hold on, wait for it, it's a beauty): "It's not my fault if they are not well enough informed to  grasp the underlying deep measurements correctly."

pfft

Good luck getting your invoice processed.
Title: Re: DxOMark vs. Reality
Post by: tron on January 15, 2013, 09:04:47 AM
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
+1000000 Very true. Rumor has it that a camera does not become worse simply because newer models appear  ;D  ;D  ;D
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 09:14:43 AM
Rumor has it that a camera does not become worse simply because newer models appear

Is that a CR1 or CR2 rumor?   ;)
Title: Re: DxOMark vs. Reality
Post by: tron on January 15, 2013, 10:30:33 AM
Rumor has it that a camera does not become worse simply because newer models appear

Is that a CR1 or CR2 rumor?   ;)
Definitely a CR3 one  ;D
Title: Re: DxOMark vs. Reality
Post by: RLPhoto on January 15, 2013, 10:52:22 AM
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
+1000000 Very true. Rumor has it that a camera does not become worse simply because newer models appear  ;D  ;D  ;D

ITS OVER NINE THOUSAND!  :o
Title: Re: DxOMark vs. Reality
Post by: paul13walnut5 on January 15, 2013, 11:22:26 AM
@bdunbar79
Quote
Well it's like me and basketball.  In HS I couldn't miss in practice from the 3 point arc.  So you could say I was a great 3-point shooter.  However, I never hit a 3 in a game because I never got open and wasn't even quick enough to get open.  So it was actually meaningless.

We here on the forum know that this is just a sensor score.  Howver, consumers don't know that and if they just read the scores they automagically assume it is a better camera.  That's the problem.  It isn't.  Sensor is just ONE measured aspect of a camera and as a scientist, when I read their crap, it comes off to me as rather unscientific.  Whether it is or isn't, it just comes off that way to me.

You can defend it all you want, but it looks like crap to me and I don't use their scores at all.  Each to their own.

Do consumers care?  When I bought my first camera I bought a Canon.  Why?  Because thats what my Dad had used for years, and what my sister used.  I think the majority of consumers are the same. 

Now I'm a bit more serious about it I read various reviews, get hands on, take in a memory card, try a couple of lenses.  I'm not very scientific, and don't photograph charts for my test, or in fact for my work or pleasure.
Maybe I could be more demanding, but I just want a camera that fits my lenses, does decent video and takes decent pics.

My cameras do that for me, I might help out a little with grading or RAW processing now and then, but I'm not losing any sleep what so ever over what Nikon or DxO are doing.   I can understand folk at the very top of their game chasing that extra 1% to the nth degree, or whatever.  I'm not at the top of my game, and I doubt that most of us here are. 

One of my favourite photographers, Maritn Parr, did a lot of his best work on a Nikon F90.  He was asked once what kind of lens he used and he had to look at the front of it to see. I mean that as no sleight.  I think we just lose track of what is important sometimes.  I think this obsession with DxO is psuedo-autistic, and as we are all churning out great images on Canon gear, can deduce it doesn't really mean that much.

I've always thought of DxO as being like reviewing a car that doesn't have a gearbox.  I've never seen a difference on a print or on screen that convinces me Canon is as far behind Nikon as the fanboys interpretation of the stats would have you believe.

If I was in the market for a 36MP camera, then I might buy a D800, I'd certainly consider it.  But I'm not.  To be honest if I could get a DSLR that did video with a stills resolution that permitted straight digital sampling to 1080 (so say a 3840 wide sensor) I would be absolutely delighted.  On that basis I will be delighted to see what DxO make of the c100 or c300, as one can only image they'll hate it!
Title: Re: DxOMark vs. Reality
Post by: Aglet on January 15, 2013, 11:58:08 AM
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
it was not when it was released
it is not much better after all the firmware updates
it made a big splash being the first CHEAP full-frame and those who wanted or needed that were so enthused with this new toy they paid little regard to its IQ shortcomings because it offered IQ benefits and features previously unavailable.

Endlessly trumpeting on that basis suckered me in to buying one; my most regrettable Canon purchase.
not for lack of due diligence, but for lack of honest and clear information about the 5D2's weaknesses, which became clear enough after I used it for a while. .. and after others began posted about its less than ideal low ISO FPN issues.

If you still find it meets your needs you either lucked out with a good one or you don't mind crushing your blacks a few more levels than some of us.

edit adding DxOmark comment below:

and THIS is what annoys me about DxOmarks results. not just that they assign a vague overall score to a camera, based solely on measured sensor merits, but that they do not adequately disclose the testing criteria and data in a way that would allow the technically astute reader the opportunity to evaluate the data on their own.  And, for the most part that I've found, neither do other sensor tech sites.

When FPN affects a sensor to the degree some of the Canon's (and some other mfr's products) were affected, it would have been very valuable to have a good idea whether the noise was random and acceptable or whether it was patterned and possibly objectionable to prospective purchasers.

DxOmark's data is useful but incomplete and that makes it much less useful than it would have been in my particular instance.
Title: Re: DxOMark vs. Reality
Post by: RLPhoto on January 15, 2013, 12:06:08 PM
How come the 5D2 IQ was impeccable before the D800? I find it still impeccable today. :|
it was not when it was released
it is not much better after all the firmware updates
it made a big splash being the first CHEAP full-frame and those who wanted or needed that were so enthused with this new toy they paid little regard to its IQ shortcomings because it offered IQ benefits and features previously unavailable.

Endlessly trumpeting on that basis suckered me in to buying one; my most regrettable Canon purchase.
not for lack of due diligence, but for lack of honest and clear information about its weaknesses, which became clear enough after I used it for a while. .. and after others began posted about its less than ideal low ISO FPN issues.

If you still find it meets your needs you either lucked out with a good one or you don't mind crushing your blacks a few more levels than some of us.

*Facepalm

If your ultra-mega analytical about bringing up black's 4-stops, which by the way is why they are called black's not midtones, then you have some other serious issues to contend with. (IE: Timing for Ideal Light)

I'd have 7D landscapes that are heavily processed and pushed to +3 in the corners than were accepted for Istock. Thats some very strict standards.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 12:35:47 PM
Last time you mentioned this "low ISO FPN issue" I did what you suggested and did a Google search for images, the only examples that search threw up were poorly exposed skies. Now I print big with a 1Ds MkIII, essentially the same sensor as the 5D MkII, and I have never had your "issue" I really want to see examples of what you are shooting, and how you are shooting it, that best illustrates your experiences.

I, too, would be interested in seeing examples of the low ISO FPN issue in real-world images, ones that don't require 100% crops and have descriptions like, "Canon 5D Mark II Fixed Pattern Noise, ISO 1600, +4 exposure," and, "...with Fill Light set to +100 to exaggerate the vertical banding in shadows."
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 02:19:48 PM
I think the WHAT is perfectly clear.  What's not clear is the real world benefit in common shooting situations for one photographer vs. another.   The sharper lens is an apt example - while having a sharper lens certainly offers a theoretical benefit, for many real world situations, that benefit is irrelevant because factors ranging from chosen subject (intentional softening for portraiture, a need to shoot at a very narrow aperture) to workflow/output (downsizing for web/small prints) render the 'benefit' of the increased lens sharpness moot.  For example, looking at the MTF chart of the MP-E 65mm, it's not a very sharp lens...but for the vast majority of the shots I take with that lens, a sharper version of it would not matter.

Thus, the request for real-world examples of shots that were rendered 'unusable' by the low ISO FPN of the 5DII.
Title: Re: DxOMark vs. Reality
Post by: Orangutan on January 15, 2013, 02:31:40 PM
If you do not understand WHAT you can do in the REAL world with larger DR , why still discussing it?

Let me illustrate it this way: people have used slide film for decades, and it is supposed to have limited DR.  Apparently, slide film had other qualities that, for many purposes, were more important than its limited DR, and  many people chose that compromise.  Sure, photographers would have loved slide film that had 14 stops of DR, while retaining the other qualities of slide film, but that was not available. 

The same is true for modern DSLR's: we would all love DSLR's that have 16 stops of clean DR, while retaining all the other qualities of the overall device (AF, speed, high ISO, color qualities, etc).  No one (so far as I can tell) is saying that they don't want more DR.  They're saying that, as with slide film, they have chosen their compromise until a better compromise is available.  At least, so it appears to me.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 02:50:22 PM
Neuro.  You did not answer my question, do you understand - and read again what I asked
Underexposing, head room , bring in high lights more than 4 stops above middle grey
 If you understand this we take  lesson 2

Certainly, I understand.  But no one is asking for theoretical 'lessons'.  Rather, both me and privatebydesign are asking for actual examples of images 'ruined' by the FPN of the 5DII, and some idea of the frequency at which the low ISO FPN renders real-world images unusable. 
Title: Re: DxOMark vs. Reality
Post by: RLPhoto on January 15, 2013, 02:56:59 PM
Neuro.  You did not answer my question, do you understand - and read again what I asked
Underexposing, head room , bring in high lights more than 4 stops above middle grey
 If you understand this we take  lesson 2

Certainly, I understand.  But no one is asking for theoretical 'lessons'.  Rather, both me and privatebydesign are asking for actual examples of images 'ruined' by the FPN of the 5DII, and some idea of the frequency at which the low ISO FPN renders real-world images unusable.

Yes, FPN ruins the perfect test charts of obsessive camera disorder sufferers.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 03:39:18 PM
I have shown pictures after pictures taken with my d800 and 5dmk2, I have shown how it is possible to expose and recover high lights (sun sets etc etc) and the adjust the picture by moving middle grey back and lift shadows and darker tones. I think it is time that you and others stop denying the benefit of a wide DR and where unfortunately not the Canon is the leader.

shall we take lesson 2   ?

Let's skip right to Lesson 5:
Q: What is the sum of 2 + 2?
A: You must stop denying the power of arithmetic, and realize the benefit of the D800 for simple math.

Point being, you are answering questions that aren't being asked, attributing statements/beliefs to people which are not their own, and for some reason, the D800 is always part of your answer, even if it's a complete non sequitur.

To be very clear, I am neither denying the benefit of wide DR nor suggesting that Canon is a leader in that specific aspect of sensor performance.  As I've repeatedly stated, if a wide DR were my primary need, I'd be shooting with a D800.

To reiterate, what's being asked is neither 'does the D800 have better DR than the 5DII?' nor 'can an image with wider DR be pushed harder in post than an image with less DR?'.  I'm not talking about contrived test scenarios.  I could just as easily contrive a scenario where the D800 would fail miserably (take a shot with the D800 with a Nikon lens at 5:1 magnification, for example). 

I've taken several thousand shots with a 5DII in the ISO 100-400 range, and guess how many I've rejected due to 'horrible low ISO FPN'?  Zero.

Yes, FPN ruins the perfect test charts of obsessive camera disorder sufferers.

Indeed.  It also ruins shots of QPcards on barbecues, but for some strange reason, that subject constitutes 0% of my shots.   ::)
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 04:13:30 PM
Stop then  denying the benefits of great DR,  and this BS about  "real life pictures" etc etc, it is up to peoples skills to use great DR or not and as resolution it is not bad to have.

Thanks for yet another non-answer to a not-asked question.  If your lessons are intended to be instructive as to how to obfuscate and avoid answering pointed questions, and attribute motives to people that they do not espouse, perhaps those would be better delivered on a forum devoted to politics. 
Title: Re: DxOMark vs. Reality
Post by: Aglet on January 15, 2013, 04:20:13 PM
I've requested permission of a family whose casual photos I shot last summer where FPN is noticeable on dark clothing.
I expect they'll approve my request.
It's about the last time I used the camera.  Customer was totally pleased with the results, I was not.

I've basically sidelined the 5d2 as an inadequate FF Rebel with serious FPN problems, not worthy of MY use so it's seen very little action with me.
I got the thing to do landscape and other outdoor and high DR work.  It's not been able to adequately fulfill that role, the way I want to use it, so to me it was an expensive PoS with limited ability.  It DID deliver some really great images in many situations where lighting was not challenging, but was not able to do so when and where I needed it to. At the time I was also using a 40D which showed less FPN when using similar PP.  I still like and use the 40D.
So a new 5D2 having worse raw file IQ than my older, cheaper camera was a big disappointment, to put it mildly.

Maybe 5d2 was one of the better tools available at the time, doesn't mean it was good enough.  A better tool was announced less than 3 weeks prior to the 5d2, the d90.  However I was an ignorant Canon fanboy at that time and paid little attention to the competition with all the full-frame excitement and high expectations.
Now a cheap consumer camera from a competitor nearly 2 years ago, at 1/5th the cost, grossly outperforms 5d2 in this part of the operating regime it only reinforces my opinion that the 5d2 was, and still is, a grossly overhyped bit of gear.  Despite that, the vast majority of users are happy to continue using it.  That's fine for them.
Those who are content with the shortcomings of a tool they're using are not going to convince those of us who need a better tool that it's OK!  Thanks for trying.  People on here are truly helpful and a great resource to plenty of newbies. :)

Hopefully I'll get permission to post a few samples where maybe you'll actually SEE the problem.  Possibly helping spare some avid users some unpleasant surprises that they haven't yet discovered.

You guys may need to be reminded. I'm not a Canon hater, just a disappointed long time Canon user who opened his eyes and found better tools for certain shots from the competition.
I still like using my bucket of Rebel bodies, 40D, and 60D and am looking fwd to an improved 7D Mk 2 or 70D.

@privatebydesign - i often print big too. expose for the highlights, bring up the rest with various curve tools until the image DR is compressed to a useful range for printing.  This is where the 5d2 (&7D) can fail if there was a lot of DR in the original scene.  The sensor may be similar or even the same between this and your 1 series body but there's a lot more to the rest of the electronic guts that can make the difference in FPN.  Quality is rarely skimped on the 1 series.

I got better results with my 40D, 60D, and various Rebels than what the 5d2 delivered.  And now my modern Nikon gear blows them all away whenever I need to do this this kind of processing.  Right tool for the job, but not necessarily the same tool for all jobs.

@Mikael Risedal - Thanks for knowing and understanding. :) I don't know if I've seen your real world comparison shots other than the one of a shed with interior shadows.  Apparently that does not qualify as a real world image for some people.  What if you want to spy on the contents of your neighbor's shed?  ;)

I suspect Neuro will tell me that I wasted 1 or 2 possible stops worth of DR on some of the shots by some of the method I used.  I.E.  I could have ETTR more if I were using Neutral or Faithful instead of Standard settings so my RGB playback histogram would more closely reflect raw headroom.  Yup... Might have.  I don't have to worry about this on my other bodies tho.
Title: Re: DxOMark vs. Reality
Post by: neuroanatomist on January 15, 2013, 04:42:33 PM
I suspect Neuro will tell me that I wasted 1 or 2 possible stops worth of DR on some of the shots by some of the method I used.

Highly unlikely.  Thanks for working toward sharing real-world examples, though (I don't consider a scenario intentionally chosen and designed to make a point to be 'real-world'). 

I've taken several thousand shots with a 5DII in the ISO 100-400 range, and guess how many I've rejected due to 'horrible low ISO FPN'?  Zero.

I should add, by way of maintaining objectivity and putting things in perspective: I've taken several thousand shots with a 5DII in the ISO 100-400 range, and guess how many I've rejected due to misfocusing by the less than stellar AF system of the 5DII?  Hundreds at least, likely thousands. 
Title: Re: DxOMark vs. Reality
Post by: Zlatko on January 15, 2013, 10:13:14 PM
I've taken several thousand shots with a 5DII in the ISO 100-400 range, and guess how many I've rejected due to 'horrible low ISO FPN'?  Zero.

Same here.  Three years of heavy shooting with this camera.  Zero photos rejected due to this "horrible" problem.  The 5DII has been good.

I should add, by way of maintaining objectivity and putting things in perspective: I've taken several thousand shots with a 5DII in the ISO 100-400 range, and guess how many I've rejected due to misfocusing by the less than stellar AF system of the 5DII?  Hundreds at least, likely thousands.

Again, same here.  AF could have been better.