December 19, 2014, 11:55:16 PM

Author Topic: Bias of tests  (Read 2116 times)

xps

  • Canon 6D
  • *****
  • Posts: 404
    • View Profile
Bias of tests
« on: March 30, 2013, 06:13:28 AM »
Maybe you will be  ::) your eyes, but does anybody know, how the test-magazines (online, printed) get their cameras for testing?
Are these Cameras "specially improved" by the manufacturers? Or do test magazines get "normal" cameras from the production lines?


canon rumors FORUM

Bias of tests
« on: March 30, 2013, 06:13:28 AM »

neuroanatomist

  • CR GEEK
  • ************
  • Posts: 15213
    • View Profile
Re: Bias of tests
« Reply #1 on: March 30, 2013, 09:49:36 AM »
It varies. DPR gets preproduction cameras for evaluation, TDP gets gear retail through B&H, etc.  They should state the source explicitly, IMO.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • ***********
  • Posts: 9390
    • View Profile
Re: Bias of tests
« Reply #2 on: March 30, 2013, 11:35:28 AM »
They do not get specially prepared cameras to test.  Most of them rent, borrow or purchase their cameras and lenses. 
Some of the big companies like DPR get pre-production models, but their published testing is done on a production model.
 
For the most part, the test results are very close from site to site.  However, when sites try to hang a score on the camera, that can be misleading.  A camera with a high score might not be the best for your use, the devil is in the details.
 

Most of the online reviewers state where they get their cameras, but it is not always obvious. I recall DPR thanking Glaziers Camera in downtown Seattle for loaning them some, and they also purchase the cameras that they want to keep around and have for further testing.  Michael Reichmann  gets pre-production Sony cameras, and buys or gets loaners from a Toronto Camera Store,  I purchase all of my test cameras, no one would give me one ;)
 
Many of the testers have had to exchange or  return cameras, or wait for a firmware upgrade.  There is always personal bias (also referred to as Real World Tests).  Its human nature. 
 
As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.
« Last Edit: March 30, 2013, 11:38:50 AM by Mt Spokane Photography »

Don Haines

  • Canon EF 300mm f/2.8L IS II
  • ********
  • Posts: 3580
  • Posting cat pictures on the internet since 1986
    • View Profile
Re: Bias of tests
« Reply #3 on: March 30, 2013, 12:13:39 PM »
It varies. DPR gets preproduction cameras for evaluation, TDP gets gear retail through B&H, etc.  They should state the source explicitly, IMO.

I can't remember where, but I read somewhere that DPR sometimes gets production cameras a week or two before the official release so they can do a review on them and have it ready for the official release... It probably goes without saying that there must be a non-disclosure agreement involved.
The best camera is the one in your hands

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Bias of tests
« Reply #4 on: March 30, 2013, 12:36:34 PM »
As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

Not that I disagree with this but online media are not immune to that either. DPR are the worst offender, IMO. Their reviews are nothing but infomercials now; and they used to be more independent (but less knowledgeable) in the past.

TW

  • PowerShot G1 X II
  • ***
  • Posts: 55
    • View Profile
Re: Bias of tests
« Reply #5 on: March 30, 2013, 12:54:37 PM »
As I recall, DPR was purchased by Amazon a few years back. Went squishy after that.
35+ years of Canon gear...good grief! Still have 2 FTb cameras in working order, too!

Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • ***********
  • Posts: 9390
    • View Profile
Re: Bias of tests
« Reply #6 on: March 30, 2013, 06:04:04 PM »
As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

Not that I disagree with this but online media are not immune to that either. DPR are the worst offender, IMO. Their reviews are nothing but infomercials now; and they used to be more independent (but less knowledgeable) in the past.

When DPR was getting started, they were hawking advertising as well.
http://www.dpreview.com/news/2002/04/22/stats
 
 

canon rumors FORUM

Re: Bias of tests
« Reply #6 on: March 30, 2013, 06:04:04 PM »

Radiating

  • Canon 6D
  • *****
  • Posts: 336
    • View Profile
Re: Bias of tests
« Reply #7 on: March 30, 2013, 06:23:38 PM »
Maybe you will be  ::) your eyes, but does anybody know, how the test-magazines (online, printed) get their cameras for testing?
Are these Cameras "specially improved" by the manufacturers? Or do test magazines get "normal" cameras from the production lines?

Most review sites buy their equipment or borrow it through normal market chanels. This is one of the reason why reviews tend to be so uneven and contradictory, copies of certain Canon lenses can have as much as 30% variation in performance. That's thirty percent.


Mt Spokane Photography

  • EF 50mm F 0.7 IS
  • ***********
  • Posts: 9390
    • View Profile
Re: Bias of tests
« Reply #8 on: March 30, 2013, 06:38:27 PM »
They do not get specially prepared cameras to test.  Most of them rent, borrow or purchase their cameras and lenses. 
Some of the big companies like DPR get pre-production models, but their published testing is done on a production model.
 
For the most part, the test results are very close from site to site.  However, when sites try to hang a score on the camera, that can be misleading.  A camera with a high score might not be the best for your use, the devil is in the details.
 

Most of the online reviewers state where they get their cameras, but it is not always obvious. I recall DPR thanking Glaziers Camera in downtown Seattle for loaning them some, and they also purchase the cameras that they want to keep around and have for further testing.  Michael Reichmann  gets pre-production Sony cameras, and buys or gets loaners from a Toronto Camera Store,  I purchase all of my test cameras, no one would give me one ;)
 
 
Update:  DPR does borrow test articles, there is a lot more about it here:
http://www.dpreview.com/misc/sitefaq

Many of the testers have had to exchange or  return cameras, or wait for a firmware upgrade.  There is always personal bias (also referred to as Real World Tests).  Its human nature. 
 
As for print media, they are dependent on advertising for their income, so there is a lot of pressure for them to rate the big dollar advertisers high.

Pi

  • 1D Mark IV
  • ******
  • Posts: 937
    • View Profile
    • Math and Photography
Re: Bias of tests
« Reply #9 on: March 30, 2013, 06:57:19 PM »
Most review sites buy their equipment or borrow it through normal market chanels. This is one of the reason why reviews tend to be so uneven and contradictory, copies of certain Canon lenses can have as much as 30% variation in performance. That's thirty percent.

That might  be as well 30% variation of the accuracy of the tests, not the lenses.

canon rumors FORUM

Re: Bias of tests
« Reply #9 on: March 30, 2013, 06:57:19 PM »