Who do you trust?

[traveller]

I know of the following sites that do lens reviews:

http://www.photozone.de/
http://www.dpreview.com/lensreviews/
http://www.the-digital-picture.com/
http://www.slrgear.com/reviews/index.php
http://www.lenstip.com/
http://www.photodo.com/
http://www.bythom.com/index.htm - is of potential use for third party lenses

and of course

http://www.pixel-peeper.com/ to do your own comparisons (assuming the photographer's technique is good and you can find a hi-res version). 

I often find that there are quite large differences in their resolution (sharpness -sic) tests of the same lens.  Can this be explaiined solely by sample variation, or is it down to the testing methods they use? Also, the measurements they come back with vary, from line pairs per picture height (DPR) to 'blur units' (SLR Gear). 

I would therefore like to pose the following question to members of this forum: which site do you prefer and why?

P.S. I forgot CR itself! Whilst I think the 70-200 IS II review was good, it would be nice to have some more... Please?!!

[kubelik]

I personally take all of them with a large grain of salt; not that I don't trust them persay, but as you've noted there are often quite dramatically different conclusions drawn by the different websites and I think it's equal parts due to their testing methods and to natural sample variation.

if there is one go-to site I have, it's Bryan's at The Digital Picture.  the thing I really appreciate about his reviews is he discusses issues such as build quality, AF speed and accuracy, quality of color, contrast, and bokeh, in a very thorough but approachable manner.  he also compares each lens against pretty much the gamut of its sibling lenses, which I find extremely useful.  most of his observations for any given lens pretty much are in sync with what I've noticed from my own lenses.

as for all the pixel-peeping ... I mean, you can measure blur units and line pairs all day, but there's a lot more to a lens.  to me, quality of bokeh and AF speed and accuracy are probably the most underrated aspects to a lens ... you don't hear it talked about as much though simply because it's hard to test empirically.

CR did a pretty good job on the 70-200 and 1D Mk IV reviews ... also waiting to see more

[CR Backup Admin]

users of these sites, including DXO should be aware that they do NOT test lenses, and do not have equipment capable of testing lenses.

What the sites do test is a lens / camera combination.  What this means to those who read the results carefully, and they tell you this, is that the results only apply to a lens used on that exact camera model and not on others.

I've found that test results on the different sites for the same camera / lens combination are usually very close, but if a lens is tested on a Canon 30D on one site, but a Canon 7D on another, the 7D should always have a higher resolution.

This is due to the fact that the system resolution is the product of the mtf of the individual components, and that includes lens, sensor, in-camera processor, software on your computer, the test chart, lighting, and even a monitor used to display the image.

Its no wonder that testers using different cameras and equipment get different results.  Lenstip, for example, admits that they get different test results from others because they use different charts, among other things.

However, if you pay attention and read the test procedures, and compare ones that use the same camera / lens combination, the results are almost always close.

[traveller]

users of these sites, including DXO should be aware that they do NOT test lenses, and do not have equipment capable of testing lenses.

What the sites do test is a lens / camera combination.  What this means to those who read the results carefully, and they tell you this, is that the results only apply to a lens used on that exact camera model and not on others.

Very true, Photozone also points out that it's results are not comparible across different systems.  Whilst I'm aware that a 18MP sensor is more likely to show a lens' limitations than an 8MP sensor, I'm always intrigued that some 'lens test' results are so poor compared to others.  For example, the EF 85mm f1.8 comes out looking quite poor in the SLR Gear sharpness (sic) test, but better in the Photozone test (Photozone seem to have got a better copy when they re-tested this lens at 15MP).  Perhaps it has more to do with where each reviewer sets their 'scales' in terms of what is excellent, good, acceptable and poor. 

I guess the advantage of the The Digital Picture's ISO12233 chart crops is that you can see for yourself what each lens' results look like, but even then they are only tested at one distance. 

Finally, I have to agree with kubelik; there is indeed more to a lens than its resolving power, unfortunately I have learnt this through bitter experience!

[foobar]

First of all: I'd never compare absolute numbers between different review sites since they all have different testing methods.


My go-to sites for lens reviews are photozone and the-digital-picture. I also visit most of the others listed occasionally.

Personally, I don't think that test results very drastically between sites (again, never comparing absolute numbers). The conclusions do, though. But that's because everyone has personal preferences.

Look at the 30mm Sigma for example: Some review sites praise it, others say it's a piece of junk. The test-results are always similar, though: Sharp center but extremely bad borders until stopping down heavily. Also a bunch of other (more or less) minor faults. A reviewer might dismiss this lens because of the border performance alone and I couldn't even fault him for that. On the other hand, you could say that already gives great sharpness in the center at f/1.4, better than for example the 28mm Canon at f/1.8.

It takes a bit of time to build you own preferences and therefore to be able to filter out which optical faults you could live with. For me: I do care about excessive CA since removing it requires me to do another post-processing step (even if it's just a small one). I don't care much about vignetting (unless it's really excessive, meaning 2 stops or more) because I think it actually makes image look a bit less sterile (I often even use it as an effect). I do care about center sharpness wide open (doesn't have to be "perfect", but at least "perfectly usable"). I don't care much about border sharpness at f/1.4-f/2.8. At these apertures, most of the image becomes an undefinable blur anyway. And speaking of that: I do care about the look (not the amount) of the lens' bokeh. I can't stand unwanted onion-rings and double-lines in the background of my pictures. I also do care about build quality and features. I'll probably never buy an autofocus lens without ring-USM again. I also like to have a distance scale and a metal mount. Equipment should be solid, responsive and fun to use. It doesn't need to be the most expensive for that (many of the cheaper lenses fall into this category as well).

Others might not agree with that priority list. And that's the point: It's the same with reviews and the people who write them.

[neuroanatomist]

Agreed - for me, it's the-digital-picture.com, then photozone.de, then a smattering of others (some of which, like William Castleman's reviews, are linked to on TDP's review pages.

As some have mentioned, comparing raw numbers across sites is not realistic.  But, it's also important to read many/most of the reviews on a particular site, to get a feel for the style.  For example, while I find Bryan's reviews very reliable, it's important to note that his reviews are generally positive and upbeat - he usually finds something to praise about every lens (which is not unreasonable, but helpful to know if you're just reading one review).

[CR Backup Admin]

A DPR reader once complained that a lens had a higher rating in DPR than photozone.  The DPR reviewer replied that the results were remarkably similar, its just that photozone is harsher than they are, but they are seeming the same results, just intrepreting it differently.  As long as you understand this, then differences between reviewers tend to become minimal.

I always check photozone, the digital picture, and other sites that have reviewed a lens.  If all but one agree, I can usually find a explanation for the difference.

I'm also suspicious of a site that seems to have very close connections with a lens company.  I'm not going to name it, (It is one of the ones named above) but I can't help thinking about the apparent connection when I see a high rating on that site and low ones elsewhere.  I even bought one of the high rated lenses once, it was horrible and went right back, so I tend to avoid that site.  Still, it might just be the luck of the draw that my lens was bad.  One bad lens does not really condem a company.

I had 5 Sigma lenses that would not work on Canon DSLR's, but did work on old Canon film cameras (a common story).  I  was able to get Sigma to rechip one of them (105mm f/2.8 macro) at my expense ($100).  I keep the rest that Sigma would not rechip to remind me that there is no guarantee that a 3rd party lens will work on tomorrows new DSLR.  I've never had a issue like that with Tamron or Tokina, just  older Sigma lenses, and those made by Sigma for other companies like Quantaray.  Except for the 105mm Macro, they came with old Canon film cameras that i had bought.

[MintMark]

One thing I appreciate about photozone is that they have tested a large variety of lenses in the same way on the same (resolution) body. This means that I can get an idea of how different lenses compare with each other. For example, if I think about buying an older prime lens I can see whether a more recent zoom lens can achieve the same results. I can get an idea of the improvements that a more expensive lens might offer and decide whether it's worth it.

[traveller]

I'm also suspicious of a site that seems to have very close connections with a lens company.  I'm not going to name it, (It is one of the ones named above) but I can't help thinking about the apparent connection when I see a high rating on that site and low ones elsewhere. 

Yes, I think I know which one you mean.  A certain brand seems to get their lenses highly rated with major faults (like poor AF accuracy) skimmed over.  It's always going to be difficult for a website that accepts lenses for testing from (what appears to be) their country's importer of the aforenotmentioned brand.  You get early access to new lenses at no expense to yourself, but if you give too many negative reviews the plug will be pulled.  I guess it's the same with any print magazine or website that lives off advertising. 

One good thing about Bryan Carnathan's reviews is that I always feel that he has actually used the lens 'in the field', rather than simply setting it up on a testing rig.  I think that this can often be the cause of reviewers missing important aspects of a lens' handling and behaviour. 

[blackcro55]

Digital picture for sure, but slr gear and fredmiranda.com for owner comments. Then to the store for a trial run.

[Isurus]

Like others here, for Canon gear I go to the-digital-picture.com.  I also refer to diglloyd.com for reviews and photography tips, especially for info on Zeiss lenses.  For user reviews, I pretty much solely use the Fred Miranda forums.

[ronderick]

wow... looks like I'm just reaffirming all others before me.

I got started with Digital Picture - especially the lenses.

Valuable opinions came from Fred Miranda when trying to decide on the gears.

[JRSJ]

The-Digital-Picture.com

Anyone else is a Ken Rockwell wannabe.

[tzalmagor]

I try to read as many reviews as I can, which usually include the first five sites on the list, Ken Rockwell, and a Amazon (have to sort through the reviews there, but some are gems).

I also ask on local photography forums, some of which offer alternatives and comparisons.

Finally, I go to one of the local stores with staff who knows the stuff, rather than people who might have sold TVs and stereos just as well.

[EYEONE]

The-digital-picture.com for sure.
I like DPreivew.com too but I'm a little limited by what is blocked here at work.

I also like cameralabs.com as well.