Dxo tests canon/nikon/sony 500mm's

[agierke]

DxO should come out a standard testing camera for testing lens... a mirrorless camera should be idea since it'll have the shortest flange focal distance and can use adapter for different vendor lens... then the only variables will be the lens and the result can be valid to compare between different lens.

that wouldnt really be more valid as you still have to mount those lenses on their brand camera for real world purposes. why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

not that i really care about DxO that is....

 

[Plainsman]

In defence of Dxo - at least they publish test info like MTF-50 sharpness graphs for you read and interpret as you want.

Nothing I dislike more than wishy washy "reviews" with nothing to justify the claims of the reviewer - like juzaphoto stating that the Canon 70-200/2.8 II is "pretty poor with teleconverters"! At least try another lens before you make a statement like that juza.

BTW Nikon will not like Dxo's tests which show that their expensive new 80-400 appears to be inferior to Sony's cheaper latest 70-400 offering certainly at the top end.

Anybody investing in an expensive new lens needs as much info as possible - so carry on Dxo.

Sony 500/4 more expensive than Canon 500/4 II. Ridiculous - won't tempt many pros over to their system.

 

[jthomson]

The most important factor in their BS Score is transmission, which is why the cheap 50/1.8 lenses from both Canon and Nikon score several points higher than any of these 500/4 lenses. It's only when you have lenses of identical max aperture that the other stuff has any influence. BTW, while the 500/4s score 25, the Canon 50/1.8 on a 5DIII gets 28, and the Nikon 50/1.8 on a D800 gets a 31, and main measurement difference between the 50/1.8s is that the Nikon is 1 P-Mpix sharper (put it on the D3X, it's sharpness and Score tie the Canon). So, the Nikon 50/1.8 is 1 P-Mpix sharper and gets a Score 3 points higher, the Canon 500/4 is 3 P-Mpix sharper, but the Scores are equal.

BS.

I begin to wonder if the 'secret metric' you mention is sponsorship...
[/quote]

If transmission really was the most important factor then the Sony should be out scoring the Nikon. The Sony equals or beats the Nikon in all the listed categories and has the lowest transmission of the three lenses.

Basically I think DxO are a bunch of Nikon fanboys. The individual ratings are fine, but the composite scores ae just ridiculous.

 

[Orangutan]

why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so. It will far outlast the current "best" body of a brand, possibly by several generations. If I'm going to lay out that much money for a lens, I want to know not only how well it will do on the current bodies, but get a sense of its longevity.

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible. (of course that's difficult, but that's what I'd like to see)

 

[weixing]

DxO should come out a standard testing camera for testing lens... a mirrorless camera should be idea since it'll have the shortest flange focal distance and can use adapter for different vendor lens... then the only variables will be the lens and the result can be valid to compare between different lens.

that wouldnt really be more valid as you still have to mount those lenses on their brand camera for real world purposes. why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

not that i really care about DxO that is....

Hmm... might not mean anything to Nikon DSLR user, but Canon user can mount Nikon lens to Canon DSLR...

Have a nice day.

 

[Loswr]

If transmission really was the most important factor then the Sony should be out scoring the Nikon. The Sony equals or beats the Nikon in all the listed categories and has the lowest transmission of the three lenses.

Transmission on macro scale, not micro. In fact, probably not even measured transmission, but rather the specified max aperture. That's why the 50/1.8 lenses outscore the 500/4 lenses.

If you look at the test results they are measuring the combination of lens and body. Hard to do otherwise. Makes comparison between manufactures difficult and less than meaningful.

Yes, and sorry, but I think you are missing the point. For every measurement, all generated with a body attached, the Canon lens comes out on top, in some cases by a significant margin. Yet, the Score is a tie. So...the score is fabricated, pulled from their nether orifices, etc.

If you go over to photozone.de's page, they'll quite clearly state that the measurements (or scores) from one camera system (make + sensor size) cannot be compared with another. Given this I can't see why the same wouldn't be true for DxO. What does that mean? That you cannot compare a score of 25 for the Nikon lens with a score of 25 for the Canon or 22 for the Sony lens.

To be able to compare each lens properly would require each lens being mounted on the same camera.

How do they calculate their lens score? Would love to know but I'm pretty sure that it is corporate secret. For the Nikon one to be so high must mean that the readings are somehow weight on the sensor (e.g pixel size.)

The point is that they measure several parameters of optical image quality from the lens, such as sharpness, transmission, distortion, vignetting, and CA. They could generate a Lens Score based on those parameters, but they don't. Had they done so, the Canon 500/4 II would have soundly trounced the Nikon 500/4.

For those who argue that it's reasonable that DxO consider the camera in the 'Lens Score', note that the sensor is already factored into the measurements themselves. P-Mpix measures sharpness of camera + lens, pixel size affects CA, etc. Even their transmission measurement changes with different cameras. So by factoring in the camera directly in the measurements (reasonable) then factoring it in again in the overall score where it's given an undisclosed (but evidently very significant) weighting, means their 'Lens Score' is as much if not more a camera score than a lens score.

One more point - considering just their P-Mpix measure of sharpness, by their definition the Nikon 500/4 results in a loss of more that 55% of the resolution of which the D800 sensor is capable, whereas the Canon 500/4 II only decreases the 5DIII's potential resolution by less than 14%. Of course, the Canon lens also outresolves the Nikon by an absolute assessment, even taking the higher resolution D800 sensor into account. But they get the same 'Lens Score'. Right.

They should rename their Overall Scores to a Camera Basic Score and a Lens Basic Score, so we could abbreviate them for what they really are: BS. Actually, that's probably giving them too much credit, because real Bovine Scat makes good fertilizer, whereas DxOMark's BS has no real-world utility.

 

[msm]

whereas DxOMark's BS has no real-world utility.

It does precisely what it is supposed to do, generate a lot of views for the DXO site where they advertise and sell their software. This is evident from this and other threads. It costs them very little since they need to test the gear to make their software anyways.

 

[Woody]

For those who argue that it's reasonable that DxO consider the camera in the 'Lens Score', note that the sensor is already factored into the measurements themselves. P-Mpix measures sharpness of camera + lens, pixel size affects CA, etc. Even their transmission measurement changes with different cameras...

Actually, that's probably giving them too much credit, because real Bovine Scat makes good fertilizer, whereas DxOMark's BS has no real-world utility.

Good point.

 

[Loswr]

...
They should rename their Overall Scores to a Camera Basic Score and a Lens Basic Score, so we could abbreviate them for what they really are: BS. Actually, that's probably giving them too much credit, because real Bovine Scat makes good fertilizer, whereas DxOMark's BS has no real-world utility.

Your assertions of DxO publishing BS are just as BS as their results because you don't know how they calculate their "lens score" and thus you have to make up reasons as to why it is so. Yes, you're compounding the BS by stating that they're publishing BS. Just don't buy into it.

Except I'm not scoring lenses without disclosing my methods. In my scientific publishing, I must fully disclose my methods such that another scientist can duplicate my experiment. DxOMark is clearly not held to the standards of peer-reviewed scientific publications. (Which is reasonable, as they're not publishing in scientific journals, but only to their own website - so, they can make up 'data' or 'scores' if they choose, change those data post hoc without explanation, whatever they want.)

We don't know what's in their black box? They don't say. Why not? If someone is hiding something, it's usually for a good reason. Could be as simple as maintaining a competitive advantage (against who, I have no idea).

Clearly, I don't buy into it. Sadly, I suspect many people do...

 

[Woody]

Your assertions of DxO publishing BS are just as BS as their results because you don't know how they calculate their "lens score" and thus you have to make up reasons as to why it is so.

Actually, based on their description, we have some idea of how the lens score is computed. For example, we know they take sensor score into account. However, this method of computation makes no sense.

 

[Pi]

why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so. It will far outlast the current "best" body of a brand, possibly by several generations. [...]

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible. (of course that's difficult, but that's what I'd like to see)

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

 

[jrista]

why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so. It will far outlast the current "best" body of a brand, possibly by several generations. [...]

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible. (of course that's difficult, but that's what I'd like to see)

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

MTF charts are, for all intents and purposes, "pure lens tests". They already give us a way to compare lenses across the board, brands be damned. Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses. The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

 

[Orangutan]

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

You don't; however, you do know that a better lens will perform no worse, and likely better, on any future camera than would a lens that performed worse in a "pure" lens test. Though not perfect, it's superior to the tainted lens+body test.

MTF charts are, for all intents and purposes, "pure lens tests". They already give us a way to compare lenses across the board, brands be damned.

If memory serves me well (no guarantee here) most manufacturers use computed (theoretical) MTF charts that are derived from the design of the optics. A few (?Zeiss?) actually test production copies. I hope someone will correct me if I'm mistaken. Yes, it would be great to see MTF charts using data from real, production copies.

 

[TexasBadger]

Since you can mount the Nikon lens on a Canon body with an adapter, why not compare apples to apples?

 

[sandymandy]

i didnt read it...let me guess: nikon lenses the best as always? except in sales...

 

[applecider]

This test should be added to a permanent faq so that any time a dxo discussion come up we all refer to this chart to show that the tests are meaningless only done to make nikon look good.

 

[Orangutan]

i didnt read it...let me guess: nikon lenses the best as always? except in sales...

Nope.

This comparison of DXO's results is what makes me EXTREMELY SUSPICIOUS of them. The Nikkor 500mm scores 25, when every single trait that factors into that score is worse than the Canon 500mm...which also scores 25. That is just plain wrong. The Canon has zero distortion, higher sharpness, less vignetting, less CA, and the same transmission...on a LOWER RESOLUTION BODY! It should have a higher score than the Nikon.

 

[Pi]

But there is still a problem: if you have a pure lens test, are you sure you know how to compute how it will perform on a future, say, 50mp body?

You don't; however, you do know that a better lens will perform no worse, and likely better, on any future camera than would a lens that performed worse in a "pure" lens test. Though not perfect, it's superior to the tainted lens+body test.

But a better lens on, say, the 5D3, will preform better on the 5D4, you know that for sure. Again, where is the problem exactly?

 

[Pi]

Since you can mount the Nikon lens on a Canon body with an adapter, why not compare apples to apples?

You mean, an apple on an orange to an orange on an orange?

 

[Pi]

Simple fact of the matter is a better lens will perform better on ALL sensors, 20mp, 30mp, or 50mp. The problem with DXO's tests is they quite simply don't give you a reasonable camera-agnostic basis from which to compare lenses.

Actually, they do. There is a way to extract the pure lens resolution from the data they used to publish (full MTF curves, not the nonsense they publish now).

The Nikon 500/4 performs "on par" (toung in cheek) with the Canon 500/4 solely because of the higher resolution sensor. That sort of tells you that the Canon lens is particularly good, because it is performing so well on a worse sensor...but you don't really have any exact way of comparing. You only get a "feeling" that it performs so well.

Why in the world would you want to know how a Canon compares to a Nikon without a body? For bragging rights? They tell you what is achievable with the current bodies on which the lens works, the way it is deigned to work. A better lens on one body will be better on future bodies as well.