Inconsistent reviewing of lenses

[AlanF]

We read reviews and draw our own conclusions, and generally have a feel for the sites that are reliable. This review of the 100-400mm II, however, takes the cake for contradicting itself.

http://www.photographyblog.com/reviews

Tamron 150-600
150mm
Both centre and edge sharpness are excellent stopped-down at f/8 through to f/16
250mm
Both centre and edge sharpness are excellent stopped-down at f/8 through to f/22
450mm
Both centre and edge sharpness are excellent stopped-down at f/8 through to f/22
600mm
Both centre and edges aren't critically sharp until you stop-down to f/11, which means that you should only use the 600mm focal length sparingly if sharpness is your number one priority.

Summary:
Score for IQ 4.5/5

Sigma 150-600
150mm
Centre and edge sharpness are commendably good wide-open at f/5, and remain high through from f/5.6 to f/16
300mm
Centre and edge sharpness are commendably good wide-open at f/5.6, and remain high through from f/8 to f/16
400mm
Centre sharpness is commendably good wide-open at f/6.3, and both the centre and the edge remain high through from f/8 to f/16
500mm
Centre sharpness is commendably good wide-open at f/6.3, and both the centre and the edge remain high through from f/8 to f/16.
600mm
Centre sharpness is commendably good wide-open at f/6.3, and both the centre and the edge remain high through from f/8 to f/16.

Summary:
Score for IQ 5/5

Canon 100-400 II

100mm
Both centre and edge sharpness are both excellent wide open at f/4.5, and outstanding between f/5.6 and f/11.
200mm
Both centre and edge sharpness are both excellent wide open at f/4.5, and outstanding between f/5.6 and f/22.
300mm
Both centre and edge sharpness are OK wide open at f/5, and outstanding between f/8 and f/16.
400mm
Both centre and edge sharpness are OK wide open at f/5.6, and outstanding between f/8 and f/16.

Summary
Score for IQ 4.5/5

Firstly, the 100-400 beats the other two in his own descriptions and yet gets rated lower than the Sigma and the same as the Tamron. Secondly, photozone, which actually measures the sharpness quantitatively, has the centre sharpness at 300mm close to the top of the chart, and wide open is sharper at f/5.6 than f/8 for 200-400mm.
http://www.photozone.de/canon_eos_ff/896-canon100400f4556is2?start=1

Very odd comments: “While you could use it (100-400) on a smaller APS-C body for a 160-640mm equivalent angle of view, it won't balance very well at all - as demonstrated by the images below, it's a much better match for a professional-grade full-frame camera like the 5D Mark III”.
“an excellent telephoto zoom lens for Canon full-frame DSLR owners”

But, the APS-C 7D II is only a couple of mm smaller in 2 of the dimensions and larger in the 3rd, and weighs 910g versus 950g for the 5D III. I can’t tell any balance difference between the lens on the 5D III and 7D II. The Tamron and especially the Sigma are far more unbalanced

 

[Mt Spokane Photography]

I tend to prefer actual data and properly done tests rather than my or someone else's calibrated eyeball.

I come from a technical job running tests in my lab for clients like NASA, and I'd have lost my job if I did tests like those. "Real World Targets", meaning a bookshelf with natural and varying amounts and colors of light falling on it, and then comparing other products under almost certainly varying conditions.

To properly compare products, they need to be tested under the exact came conditions, which includes light levels and colors.

A sloppy test procedure only causes me to wonder if any of the other parameters were properly controlled. Maybe the solid tripod was affected by every truck rolling by, that's real world, after all.

Most of the online lens testers are limited by funds and test equipment, but some of them at least manage to turn in consistent results.

I have no reason to say his results are right or wrong, but I do not trust the test procedure he reveals.

 

[YuengLinger]

Ok, I do think we need a policy that requires reviewers to be certified and licensed in the state or province where they reside. Furthermore, such a license should not be issued to anybody who does not have at least an MA and an inspected, state-of-the-art lab.

Facetiousness aside, we get what we pay for in reviews.

And worse, paid influencers are all over the place, even on this website, alas.

To quote Mr. Monk, "It's a jungle out there."

Thankfully, AlanF and others are sharp enough and concerned enough to spot and report poorly executed tests and reviews.

 

[surapon]

I tend to prefer actual data and properly done tests rather than my or someone else's calibrated eyeball.

I come from a technical job running tests in my lab for clients like NASA, and I'd have lost my job if I did tests like those. "Real World Targets", meaning a bookshelf with natural and varying amounts and colors of light falling on it, and then comparing other products under almost certainly varying conditions.

To properly compare products, they need to be tested under the exact came conditions, which includes light levels and colors.

A sloppy test procedure only causes me to wonder if any of the other parameters were properly controlled. Maybe the solid tripod was affected by every truck rolling by, that's real world, after all.

Most of the online lens testers are limited by funds and test equipment, but some of them at least manage to turn in consistent results.

I have no reason to say his results are right or wrong, but I do not trust the test procedure he reveals.

+100 With you, Sir, Dear Teacher, Mr. Mt Spokane.
As my Professional Architect, Yes, I can Use my Eye balls and My Dip stick to test the Strength of Ready to pour Concrete from Concrete Truck and use the Slump Test Cone to test at the job site too, BUT, I must send the poured Cylinders of 10 to test in the Lab. for the REAL LAB TEST Strength of Concrete, Before to make my decision of accept or reject that batch of concrete.
Yes, Sir, Too many Factors at the field to mess up the test by eye balls, such as the great 1200 US DOLLARS Tripods or 59 US Dollars Tripods, Or the Moisture in the air that create the mini-fog/ mini haze that interfere of the quality of Lens in that mili-seconds of the press shutter to recorded the picture.
Have a great Wednesday, Sir.
Surapon

 

[NWPhil]

Ok, I do think we need a policy that requires reviewers to be certified and licensed in the state or province where they reside. Furthermore, such a license should not be issued to anybody who does not have at least an MA and an inspected, state-of-the-art lab.

Facetiousness aside, we get what we pay for in reviews.

And worse, paid influencers are all over the place, even on this website, alas.

To quote Mr. Monk, "It's a jungle out there."

Thankfully, AlanF and others are sharp enough and concerned enough to spot and report poorly executed tests and reviews.

Haven't you heard? if it is on the web, it's true.... ;D

It's good to hear about other people experiences and feedback, but as mt Spokane is saying not everyone is qualifyed to perform reliable tests - there are only a few

 

[mackguyver]

I typically look for trends, but generally trust the sharpness maps of SLRGear and the detailed testing of LensTip and DxO. I also find that no one tests color and contrast, which are very important to me as well. When I did brick wall tests of the 24-70 I & II and 16-35 f/2.8 II & 16-35 f/4 IS, those two were huge improvements.

Ultimately, I try out the lens as some lenses like the controversial 50L suck at testing, but excel at portraits. Same goes with the 180L macro. A German site shows it as the sharpest lens they have tested, but everyone else shows it as decent. My copy is very good, but not my sharpest lens. It has the best color and contrast of any lens I own, however.

 

[dak723]

The only test that really matters is trying the lens yourself and seeing the results. If you can rent the lens (and afford the not so cheap rental fees) then that is always a good idea. If not, I would always recommend purchasing from a dealer that allows returns. Personally I always feel bad if I return an item that is not faulty and I am returning it purely because I don't like it, but in today's world - where you often buy items unseen online - that has become a necessity.

 

[cycleraw]

I own the sigma 150-600 sport as well as the canon 100-400 II. The canon is sharper, focuses much quicker and accurately than the sigma. The only advantage the sigma has is 200mm additional reach.

 

[Mt Spokane Photography]

I typically look for trends, but generally trust the sharpness maps of SLRGear and the detailed testing of LensTip and DxO. I also find that no one tests color and contrast, which are very important to me as well. When I did brick wall tests of the 24-70 I & II and 16-35 f/2.8 II & 16-35 f/4 IS, those two were huge improvements.

Ultimately, I try out the lens as some lenses like the controversial 50L suck at testing, but excel at portraits. Same goes with the 180L macro. A German site shows it as the sharpest lens they have tested, but everyone else shows it as decent. My copy is very good, but not my sharpest lens. It has the best color and contrast of any lens I own, however.

While lens tip does good testing, I'm uneasy about someone testing lenses supplied by the manufacturer. Its a really big temptation for a manufacturer to give out a lens that is at the high end of the production range. They have automated test equipment that can record test results as lenses move down the production line, and have a minimum acceptable MTF, but there are some where everything comes together just right and gives higher values.

BTW, everyone measures MTF, which is a test for contrast.

 

[mackguyver]

I typically look for trends, but generally trust the sharpness maps of SLRGear and the detailed testing of LensTip and DxO. I also find that no one tests color and contrast, which are very important to me as well. When I did brick wall tests of the 24-70 I & II and 16-35 f/2.8 II & 16-35 f/4 IS, those two were huge improvements.

Ultimately, I try out the lens as some lenses like the controversial 50L suck at testing, but excel at portraits. Same goes with the 180L macro. A German site shows it as the sharpest lens they have tested, but everyone else shows it as decent. My copy is very good, but not my sharpest lens. It has the best color and contrast of any lens I own, however.

While lens tip does good testing, I'm uneasy about someone testing lenses supplied by the manufacturer. Its a really big temptation for a manufacturer to give out a lens that is at the high end of the production range. They have automated test equipment that can record test results as lenses move down the production line, and have a minimum acceptable MTF, but there are some where everything comes together just right and gives higher values.

BTW, everyone measures MTF, which is a test for contrast.

Lenstip used to review lenses loaned to them by regular people, but maybe that's not the case anymore and if so, that's a good point. Also, I agree on the MTF part, though there are two parts to resolution (contrast and resolving power) and the reviewers nearly always comment on the resolving power.

Ultimately using the lens yourself is the only way to decide if it's good enough for your needs. I have been disappointed by some well-reviewed lenses and amazed by some that have been poorly reviewed. Speaking of which, I think it's time to borrow something new from Canon as part of my CPS membership

 

[slclick]

2 sources help me make decisions

TDP and Lens Rentals. Anywhere else always leaves me scratching my head.

 

[monkey44]

One of the most important tests for me personally, beyond the independent 'calibrated measurements' of each element that we can test fall outside the labs. Put that baby in my hands and let me see what it does when we assemble all the individual components into a camera / lens to actually capture an image when all those lab controls are absent.

A lab can control all it wants, and give us a great overview of the potential, but we don't meet those lab controls often in the field -- that's where the most important tests take place.

I've used cameras where "some" of the reviewers or labs scream 'this camera won't do this, or won't do that', and then gone out and done it. LIGHT!!! That's ALWAYS the issue, and in other than studio conditions, we cannot control it easily other than choose the time and space we inhabit when we pull that trigger.

Lab testing gives us great info on POTENTIAL, but shooting in the field is always the best and final review. It's also based in technique and experience as well, which can often overcome adverse field conditions (well, as often not too)

 

[AlanF]

Testing continues to amaze me. ePhotozine has just published their measured mtfs of the 100-400 II on the 5DIII and have quite incredibly at 400mm the following, with a sharp increase at f/8 and f/5.6, f/11 and f/16 the same Top). Do abrupt peaks like that rea;lly happen (top graph). On the other hand, photozone.de have a smooth transition with f number (bottom), and TDP has f5.6 ~ f/8 and f/5.6 clearly better than f/16
http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=972&Camera=453&Sample=0&FLI=4&API=1&LensComp=972&CameraComp=453&SampleComp=0&FLIComp=4&APIComp=3

Perplexed.

 

[Rahul]

Couldn't the different results be because of copy variation?

Personally, the most believable test results / reviews are from Lensrentals who test multiple copies while also explain the methodology of the test procedure.

 

[AlanF]

Couldn't the different results be because of copy variation?

Personally, the most believable test results / reviews are from Lensrentals who test multiple copies while also explain the methodology of the test procedure.

I don't think so. It simply looks inconsistent that there is a radical jump in MTF from f/5.6 to f/8 and then back again. Also the flat response from centre to edge looks very odd - even Canon's MTFs show the centre best and the edges worse. If I had made measurements like these in one of my experiments, I would be checking them.

 

[c.d.embrey]

“While you could use it (100-400) on a smaller APS-C body for a 160-640mm equivalent angle of view, it won't balance very well at all - as demonstrated by the images below,"

Why would I trust anyone who doesn't even know how to hold a camera/long lens ??? I've used a Film Rebel with 300m f/2.8 and a 20D with a 100-400, both with NO Problems.

BTW my favorite Real World tester shoots cereal boxes in his kitchen : as part of his lens tests.

 

[AlanF]

I decided to do my own tests, staring pixel by pixel at shots at f/5.6 and f/8 with the 100-400 II on the 5DIII and 7DII of the iso12233 chart and the FoCal chart at 400mm. This was important for me as I want to know the best aperture. On both cameras, f5.6 is slightly sharper for resolving closely spaced lines etc. Also, the bare 100-400 on the 7DII is marginally better than +1.4xTCIII on the 5DIII. The superiority of the f/5.6 setting cannot be due to an error in AFMA as f/8 would have a greater depth of field. Perhaps there is some variation between copies, but I am happy to have one that is best wide open.