Bias of tests

[xps]

Maybe you will be : your eyes, but does anybody know, how the test-magazines (online, printed) get their cameras for testing?
Are these Cameras "specially improved" by the manufacturers? Or do test magazines get "normal" cameras from the production lines?

 

[Loswr]

It varies. DPR gets preproduction cameras for evaluation, TDP gets gear retail through B&H, etc. They should state the source explicitly, IMO.

 

[Mt Spokane Photography]

They do not get specially prepared cameras to test. Most of them rent, borrow or purchase their cameras and lenses.
Some of the big companies like DPR get pre-production models, but their published testing is done on a production model.

For the most part, the test results are very close from site to site. However, when sites try to hang a score on the camera, that can be misleading. A camera with a high score might not be the best for your use, the devil is in the details.


Most of the online reviewers state where they get their cameras, but it is not always obvious. I recall DPR thanking Glaziers Camera in downtown Seattle for loaning them some, and they also purchase the cameras that they want to keep around and have for further testing. Michael Reichmann gets pre-production Sony cameras, and buys or gets loaners from a Toronto Camera Store, I purchase all of my test cameras, no one would give me one

Many of the testers have had to exchange or return cameras, or wait for a firmware upgrade. There is always personal bias (also referred to as Real World Tests). Its human nature.

As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

 

[Don Haines]

It varies. DPR gets preproduction cameras for evaluation, TDP gets gear retail through B&H, etc. They should state the source explicitly, IMO.

I can't remember where, but I read somewhere that DPR sometimes gets production cameras a week or two before the official release so they can do a review on them and have it ready for the official release... It probably goes without saying that there must be a non-disclosure agreement involved.

 

[Pi]

As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

Not that I disagree with this but online media are not immune to that either. DPR are the worst offender, IMO. Their reviews are nothing but infomercials now; and they used to be more independent (but less knowledgeable) in the past.

 

[TW]

As I recall, DPR was purchased by Amazon a few years back. Went squishy after that.

 

[Mt Spokane Photography]

As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

Not that I disagree with this but online media are not immune to that either. DPR are the worst offender, IMO. Their reviews are nothing but infomercials now; and they used to be more independent (but less knowledgeable) in the past.

When DPR was getting started, they were hawking advertising as well.
http://www.dpreview.com/news/2002/04/22/stats

 

[Radiating]

Maybe you will be : your eyes, but does anybody know, how the test-magazines (online, printed) get their cameras for testing?
Are these Cameras "specially improved" by the manufacturers? Or do test magazines get "normal" cameras from the production lines?

Most review sites buy their equipment or borrow it through normal market chanels. This is one of the reason why reviews tend to be so uneven and contradictory, copies of certain Canon lenses can have as much as 30% variation in performance. That's thirty percent.

 

[Mt Spokane Photography]

They do not get specially prepared cameras to test. Most of them rent, borrow or purchase their cameras and lenses.
Some of the big companies like DPR get pre-production models, but their published testing is done on a production model.

For the most part, the test results are very close from site to site. However, when sites try to hang a score on the camera, that can be misleading. A camera with a high score might not be the best for your use, the devil is in the details.


Most of the online reviewers state where they get their cameras, but it is not always obvious. I recall DPR thanking Glaziers Camera in downtown Seattle for loaning them some, and they also purchase the cameras that they want to keep around and have for further testing. Michael Reichmann gets pre-production Sony cameras, and buys or gets loaners from a Toronto Camera Store, I purchase all of my test cameras, no one would give me one


Update: DPR does borrow test articles, there is a lot more about it here:
http://www.dpreview.com/misc/sitefaq

Many of the testers have had to exchange or return cameras, or wait for a firmware upgrade. There is always personal bias (also referred to as Real World Tests). Its human nature.

As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

 

[Pi]

Most review sites buy their equipment or borrow it through normal market chanels. This is one of the reason why reviews tend to be so uneven and contradictory, copies of certain Canon lenses can have as much as 30% variation in performance. That's thirty percent.

That might be as well 30% variation of the accuracy of the tests, not the lenses.