September 02, 2014, 07:08:30 PM

Author Topic: Dxo tests canon/nikon/sony 500mm's  (Read 15561 times)

Woody

  • 6D
  • *****
  • Posts: 534
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #30 on: July 14, 2013, 11:16:52 AM »
DXO never said that they were "lens tests" only, so I do not see he problem.

I was surprised by your comment, so I decided to investigate the definition of DXOMark Score. This is what is said on http://www.dxomark.com/index.php/Publications/DxOMark-Insights/DxOMark-Score/DxOMark-Score-design:

"DxOMark Score can be interpreted as the maximum print size of the average quality. Obviously, any photo can be printed at any size, but beyond a certain point, a larger print does not reveal any additional details to an observer at close distance...

While the Sensor Overall Score describes the results of measurements only on sensors and is essentially related to image noise (for example, a difference of one f-stop offsets the Overall Sensor Score by approximately 15 points), the DxOMark Score is both proportional to resolution (taking optical aberrations into account) and to sensor dynamic range."

It appears that DXOMark considers both sensor and lens performances when assigning a score. So, I guess you are right.

But I still can't get over the fact the Canon 500 mm lens when mounted on a 22 MP FF sensor can OUTRESOLVE the Nikon 500 mm lens mounted on a 36 MP FF sensor. Wow, just wow! Clearly shows how awful the new Nikon telephoto lens is... especially when these prices are taken into account.

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #30 on: July 14, 2013, 11:16:52 AM »

iMagic

  • Rebel T5i
  • ****
  • Posts: 101
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #31 on: July 14, 2013, 11:41:41 AM »

[/quote]

DXO never said that they were "lens tests" only, so I do not see he problem.
[/quote]

Uh, the title says "lens reviews". Symantics and misunderstanding? When i see lens reviews i would expect apples to apples comparison and a "lens test" tone to the article. As it stands, the whole article is meaningless.


"Nikon AF-S Nikkor 500mm and 600mm f/4G ED VR lens reviews: legendary performers in the range"

bdunbar79

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2563
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #32 on: July 14, 2013, 12:37:46 PM »
If you look at the test results they are measuring the combination of lens and body.  Hard to do otherwise.  Makes comparison between manufactures difficult and less than meaningful.

Oh, I know. I've long stated that DXO's "lens tests" are really "camera system tests".

DXO never said that they were "lens tests" only, so I do not see he problem.

It's a problem because it's a lens score and nobody cares about a 5D3/500 f/4L II combo score.  They want to know which lens is better.  And, DxOmark scores it as a lens score, so an average observer would take it at face value.  It's deceiving, and an objective reviewing body should never be deceiving, if they are being truly objective.  Clearly they are not.
2 x 1DX
Big Ten, GLIAC, NCAC

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #33 on: July 15, 2013, 02:05:44 AM »
But I still can't get over the fact the Canon 500 mm lens when mounted on a 22 MP FF sensor can OUTRESOLVE the Nikon 500 mm lens mounted on a 36 MP FF sensor. Wow, just wow! Clearly shows how awful the new Nikon telephoto lens is... especially when these prices are taken into account.

DXO decided to dumb down their lens+body measurements and to report the results in some metric that they keep secret. From other discussions here, I can guess that it is heavily weighed towards higher MTF's, i.e., it measures mainly what we call "contrast". Then 36 or 22 mp does not matter much.

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #34 on: July 15, 2013, 02:08:06 AM »

Uh, the title says "lens reviews". Symantics and misunderstanding? When i see lens reviews i would expect apples to apples comparison and a "lens test" tone to the article. As it stands, the whole article is meaningless.

"Nikon AF-S Nikkor 500mm and 600mm f/4G ED VR lens reviews: legendary performers in the range"

You cannot judge an article by its title only. The text clearly talks about combos. Also, you cannot get "pure lens" data anywhere on their site, you always choose a body.

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #35 on: July 15, 2013, 02:15:46 AM »
It's a problem because it's a lens score and nobody cares about a 5D3/500 f/4L II combo score.  They want to know which lens is better.  And, DxOmark scores it as a lens score, so an average observer would take it at face value.  It's deceiving, and an objective reviewing body should never be deceiving, if they are being truly objective.  Clearly they are not.

I disagree. I would say that nobody cares how the 500/4 II performs on the D800, but how I performs on the 5d3 is really interesting. Even if DXO were to publish pure resolution numbers of the lens, nobody would know what to do with those numbers to see how it performs on a particular body (well, some would know).

Their "score" is nonsense, but the number for each aperture and FL (for zooms) are useful.

mb66energy

  • 7D
  • *****
  • Posts: 355
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #36 on: July 15, 2013, 04:51:02 AM »
The Nikon 500mm rules @ f/32 in terms of actuance:
This might lead to a higher summed up score because DxO produces the score from data over the whole range. If this is NOT WEIGHTED with typical use scenarios ... this gives misleading scores while the measurements are correct AND helpful.

This might explain why Canons and Nikons 4/500 have the same scores - I would choose Canon's lens because it should deliver cleaner results @ f/4-f/8 - the typical f-stops I would use.

I'd like to ask where DxOMark states the Score is summed over the whole range...because instead, I read: "DxOMark Score corresponds to an optimal focal length/aperture combination. The score corresponds to the quantity of information that can be captured by the camera. Each focal length/aperture combination provides a numerical value. The highest value is the DxOMark Score.". They're saying the score is based on the one best focal length/aperture combo. They indicate that right next to their Score, and for all three lenses that optimum is 500mm f/4.

Sorry, but I believe actuance at f/32 is irrelevant to the Scores, which are still BS...

I do not find the text I BELIEVE I have read - perhaps I should read english texts more carefully because it isn't my mother language. Perhaps I have misread the explanation of the METRIC tests which averages the results of different FOCAL LENGTHS - not different apertures.
Sorry for that.

Now I have seen the Use Case Scores (Mid-Light-Score) - whatever that is (clicking on the "?" doesn't provide information):
Nikon 48  @f/5.6
Canon 48  @f/4.0
Sony 44  @f/4.0
perhaps that is a weighted version of scoring ...
EDIT(added): http://www.dxomark.com/index.php/Lenses/Compare-Camera-Lenses/Compare-lenses/%28lens1%29/1174/%28lens2%29/393/%28lens3%29/891/%28brand1%29/Nikkor/%28camera1%29/792/%28brand2%29/Canon/%28camera2%29/795/%28brand3%29/Sony/%28camera3%29/831

But perhaps you are right: The scores ARE BS ...
TOOLS: EF-S 10-22 | 60 || EF 2.8/24 | 2.8/40* | 2.8 100 Macro* |2.0/100 | 4.0/70-200* | 5.6/400* || 2 x 40D | 600D | EOS M  [* most used lenses]

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #36 on: July 15, 2013, 04:51:02 AM »

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14042
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #37 on: July 15, 2013, 05:11:22 AM »
DXO decided to dumb down their lens+body measurements and to report the results in some metric that they keep secret. From other discussions here, I can guess that it is heavily weighed towards higher MTF's, i.e., it measures mainly what we call "contrast". Then 36 or 22 mp does not matter much.

The most important factor in their BS Score is transmission, which is why the cheap 50/1.8 lenses from both Canon and Nikon score several points higher than any of these 500/4 lenses. It's only when you have lenses of identical max aperture that the other stuff has any influence.  BTW, while the 500/4s score 25, the Canon 50/1.8 on a 5DIII gets 28, and the Nikon 50/1.8 on a D800 gets a 31, and main measurement difference between the 50/1.8s is that the Nikon is 1 P-Mpix sharper (put it on the D3X, it's sharpness and Score tie the Canon). So, the Nikon 50/1.8 is 1 P-Mpix sharper and gets a Score 3 points higher, the Canon 500/4 is 3 P-Mpix sharper, but the Scores are equal.

BS.

I begin to wonder if the 'secret metric' you mention is sponsorship...   :o
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

weixing

  • EOS M2
  • ****
  • Posts: 242
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #38 on: July 15, 2013, 06:41:55 AM »
Hi,
   Lens performance result using measurement from different camera body is basically useless since every camera sensor perform differently and every vendor process their RAW file differently.

   DxO should come out a standard testing camera for testing lens... a mirrorless camera should be idea since it'll have the shortest flange focal distance and can use adapter for different vendor lens... then the only variables will be the lens and the result can be valid to compare between different lens.

   Have a nice day.

Woody

  • 6D
  • *****
  • Posts: 534
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #39 on: July 15, 2013, 07:45:08 AM »
Lens performance result using measurement from different camera body is basically useless since every camera sensor perform differently and every vendor process their RAW file differently.

DxO should come out a standard testing camera for testing lens... a mirrorless camera should be idea since it'll have the shortest flange focal distance and can use adapter for different vendor lens... then the only variables will be the lens and the result can be valid to compare between different lens.

But they won't do that because a truly impartial score is not acceptable to them. ;)

agierke

  • 7D
  • *****
  • Posts: 347
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #40 on: July 15, 2013, 08:52:39 AM »
Quote
   DxO should come out a standard testing camera for testing lens... a mirrorless camera should be idea since it'll have the shortest flange focal distance and can use adapter for different vendor lens... then the only variables will be the lens and the result can be valid to compare between different lens.

that wouldnt really be more valid as you still have to mount those lenses on their brand camera for real world purposes. why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

not that i really care about DxO that is....
5D3, 5D2, 5DC, s15mm Fish, 24mm TSE, 35mm F1.4L, 50mm F1.2L, 85mm F1.8, 100mm F2.8L, 24-70mm F2.8L, 70-200mm F2.8L, 580EX, 580EX2, 600EXRT

Plainsman

  • Rebel SL1
  • ***
  • Posts: 88
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #41 on: July 15, 2013, 09:19:18 AM »
In defence of Dxo - at least they publish test info like MTF-50 sharpness graphs for you read and interpret as you want.

Nothing I dislike more than wishy washy "reviews" with nothing to justify the claims of the reviewer - like juzaphoto stating that the Canon 70-200/2.8 II is "pretty poor with teleconverters"! At least try another lens before you make a statement like that juza.

BTW Nikon will not like Dxo's tests which show that their expensive new 80-400 appears to be inferior to Sony's cheaper latest 70-400 offering certainly at the top end.

Anybody investing in an expensive new lens needs as much info as possible - so carry on Dxo.

Sony 500/4 more expensive than Canon 500/4 II. Ridiculous - won't tempt many pros over to their system.

dilbert

  • Canon EF 300mm f/2.8L IS II
  • *******
  • Posts: 2833
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #42 on: July 15, 2013, 09:32:49 AM »
If you look at the test results they are measuring the combination of lens and body.  Hard to do otherwise.  Makes comparison between manufactures difficult and less than meaningful.

Yes, and sorry, but I think you are missing the point.  For every measurement, all generated with a body attached, the Canon lens comes out on top, in some cases by a significant margin.  Yet, the Score is a tie.  So...the score is fabricated, pulled from their nether orifices, etc.

If you go over to photozone.de's page, they'll quite clearly state that the measurements (or scores) from one camera system (make + sensor size) cannot be compared with another. Given this I can't see why the same wouldn't be true for DxO. What does that mean? That you cannot compare a score of 25 for the Nikon lens with a score of 25 for the Canon or 22 for the Sony lens.

To be able to compare each lens properly would require each lens being mounted on the same camera.

How do they calculate their lens score? Would love to know but I'm pretty sure that it is corporate secret. For the Nikon one to be so high must mean that the readings are somehow weight on the sensor (e.g pixel size.)

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #42 on: July 15, 2013, 09:32:49 AM »

jthomson

  • EOS M2
  • ****
  • Posts: 150
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #43 on: July 15, 2013, 11:17:55 AM »


The most important factor in their BS Score is transmission, which is why the cheap 50/1.8 lenses from both Canon and Nikon score several points higher than any of these 500/4 lenses. It's only when you have lenses of identical max aperture that the other stuff has any influence.  BTW, while the 500/4s score 25, the Canon 50/1.8 on a 5DIII gets 28, and the Nikon 50/1.8 on a D800 gets a 31, and main measurement difference between the 50/1.8s is that the Nikon is 1 P-Mpix sharper (put it on the D3X, it's sharpness and Score tie the Canon). So, the Nikon 50/1.8 is 1 P-Mpix sharper and gets a Score 3 points higher, the Canon 500/4 is 3 P-Mpix sharper, but the Scores are equal.

BS.

I begin to wonder if the 'secret metric' you mention is sponsorship...   :o
[/quote]

If transmission really was the most important factor then the Sony should  be out scoring the Nikon.  The Sony equals or beats the Nikon in all the listed categories and has  the lowest transmission of the three lenses.

Basically I think DxO are a bunch of Nikon fanboys.  The individual ratings are fine, but the composite scores ae just ridiculous.

Orangutan

  • 5D Mark III
  • ******
  • Posts: 666
    • View Profile
Re: Dxo tests canon/nikon/sony 500mm's
« Reply #44 on: July 15, 2013, 11:21:34 AM »
why would i care what the test results of a lens would be mounted to a camera i would never shoot. i would rather see test results from a lens/body combo that i could actually use.

Because lenses, especially one like this, will probably be used on new bodies for the next 10 years or so.  It will far outlast the current "best" body of a brand, possibly by several generations.  If I'm going to lay out that much money for a lens, I want to know not only how well it will do on the current bodies, but get a sense of its longevity.

So my answer is yes: to the greatest degree feasible, a "lens test" should isolate the lens, even mounting the competitors on the same body if possible.  (of course that's difficult, but that's what I'd like to see)

canon rumors FORUM

Re: Dxo tests canon/nikon/sony 500mm's
« Reply #44 on: July 15, 2013, 11:21:34 AM »