September 03, 2014, 02:59:20 AM

Author Topic: Biased Reviewing & the consequences.  (Read 5162 times)

Flake

  • Guest
Biased Reviewing & the consequences.
« on: July 14, 2011, 11:36:02 AM »
Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading.  This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.

For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.

canon rumors FORUM

Biased Reviewing & the consequences.
« on: July 14, 2011, 11:36:02 AM »

UncleFester

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #1 on: July 14, 2011, 12:18:30 PM »
Are you talking about the beta-testers that furnish a review?

Or someone who just bought one and wrote a review.

When I look for reviews I'm always looking for the most negative. They are usually more realistic.

Canihaspicture

  • Rebel SL1
  • ***
  • Posts: 92
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #2 on: July 14, 2011, 12:41:23 PM »
I used a P7000 on a photo walk and it was garbage with a battery. However, many of the people on the same walk loved it. I'm not sure why, maybe they weren't in manual mode. It could be like an Apple brainwash thing like in the Cult of Steve Jobs, but only for Nikon.

The camera couldn't accomplish basic camera functions without problems. I'm not even mentioning the speed of the thing yet.

Personally, I like DigitalRev on youtube for reviews and commentary. That dude tells it how it is and he's funny. He is an admitted Nikon guy and even he says the P7000 sucks.

Flake

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #3 on: July 14, 2011, 12:42:28 PM »
No not beta testers, the so called more respectable ones - magazines and certain websites.  For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I just think it's so unfair of reviewers to hype a product, and then have to read people wondering why they can't get the results the reviewer has claimed.

Mt Spokane Photography

  • Canon EF 600mm f/4L IS II
  • ********
  • Posts: 8490
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #4 on: July 14, 2011, 01:05:26 PM »
Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading.  This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.

For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.


Anyone who does not agree with me is biased!

That is just human nature.  We often see people talking about using real world images to test a camera, which, in fact, can be unintentionally selected to hide defects or make them less apparent.  There are often no established tests for cameras, so opinions are all we have.  For example, we hear that AF is fast or it is slow with a lens.  How do you measure it?  Its a function of lens, body, subject lighting and contrast, and probably more.

As someone who has had to conduct lab testing for use by some very technical and particular customers, I eventually learned to separate facts and opinions in my conclusions, and to justify my opinions.  I have had  to hold back a lot of opinions because there was no hard evidence to justify it.

We all see things thru a filter which is based on our experiences.  If I've had a issue with a product in the past, I tend to be unforgiving when it comes to reviewing it.  for example, I have had four sigma lenses that were made to be EF compatible, but did not work on DSLR's because Sigma did not understand the Canon lens communication adequately.  Even now, I avoid Sigma, and when I see comments about poor autofocus accuracy, it just reinforces my experiences.

UncleFester

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #5 on: July 14, 2011, 01:12:40 PM »
No not beta testers, the so called more respectable ones - magazines and certain websites.  For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I just think it's so unfair of reviewers to hype a product, and then have to read people wondering why they can't get the results the reviewer has claimed.

I don't think a lot of reviews - magazine, consumer, some websites - are as thorough as, say, DPreview. Meaning there are probably more "knee-jerk" reviews with very little effort, given the images some present as evidence of their research - for example, leaning out the office window and shooting the buildings across the street. hardly realistic examples.

Same goes for retail reviewers.

steven63

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #6 on: July 14, 2011, 01:31:50 PM »

DPreview all the way.

canon rumors FORUM

Re: Biased Reviewing & the consequences.
« Reply #6 on: July 14, 2011, 01:31:50 PM »

Flake

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #7 on: July 14, 2011, 02:38:32 PM »
Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading.  This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.


For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.


Anyone who does not agree with me is biased!

That is just human nature.  We often see people talking about using real world images to test a camera, which, in fact, can be unintentionally selected to hide defects or make them less apparent.  There are often no established tests for cameras, so opinions are all we have.  For example, we hear that AF is fast or it is slow with a lens.  How do you measure it?  Its a function of lens, body, subject lighting and contrast, and probably more.

As someone who has had to conduct lab testing for use by some very technical and particular customers, I eventually learned to separate facts and opinions in my conclusions, and to justify my opinions.  I have had  to hold back a lot of opinions because there was no hard evidence to justify it.

We all see things thru a filter which is based on our experiences.  If I've had a issue with a product in the past, I tend to be unforgiving when it comes to reviewing it.  for example, I have had four sigma lenses that were made to be EF compatible, but did not work on DSLR's because Sigma did not understand the Canon lens communication adequately.  Even now, I avoid Sigma, and when I see comments about poor autofocus accuracy, it just reinforces my experiences.


I'd agree with all of that, but one of the problems has been with noise, where some reviewers have claimed that the D7000 is as good as the D3 when it clearly is nothing like as good.  Opinion is one thing, however exaggeration is something else, and is misleading.  One of the complaints has been from a user trying to replicate low light results & then wondering what he's doing wrong when it's nothing like as good as the review had led him to believe.

If it was just a few reviewers opinions I'd never have posted the thread, however this is blatent over hyping a product which is adversly affecting the product experience of users, dissapointing them greatly.

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14047
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #8 on: July 14, 2011, 03:12:39 PM »
I'd agree with all of that, but one of the problems has been with noise, where some reviewers have claimed that the D7000 is as good as the D3 when it clearly is nothing like as good.  Opinion is one thing, however exaggeration is something else, and is misleading.

As a scientist, I like numbers.  They don't glow, they don't hype.   DxOMark reports the low-light sensitivity, defined as the highest ISO sensitivity to which your camera can be set while maintaining a high quality, low-noise image (based on a Signal-to-Noise-Ratio [SNR] of 30dB, a dynamic range of 9EVs and a color depth of 18bits), as ISO 1167 for the D7000 and ISO 2290 for the D3, so the D3 over a stop better in low light.
 
No not beta testers, the so called more respectable ones - magazines and certain websites.  For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I'm curious as to how Digital Picture is biased, in your opinion?  I do see that the reviews there are 'rose-colored' in the sense that the author never comes right out and says something like, "This lens is a cheap piece of crap," and he finds something good to say about almost every lens (even if the IQ is low, the lens is great for a budget, etc.).  But I wouldn't call that a bias, more like a viewpoint.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

Flake

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #9 on: July 14, 2011, 04:16:53 PM »
I think digital picture is biased towards Canon products to the point where an unduely positive gloss is placed on some products and the third party brands suffer the opposite effect.  All this is fine if you know about it, but it could put someone off buying one lens if they weren't aware.

dr croubie

  • 1D X
  • *******
  • Posts: 1398
  • Too many photos, too little time.
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #10 on: July 14, 2011, 08:29:52 PM »
Just on TDP, it's a canon review site. he only reviews canon. any time he mentions nikon he says he doesn't own/use much nikon, if any.

For a canon vs canon vs canon-fit-3rd-party, it's great. every review he compares each lens to a few others in the same price/size/zoom range category, and explains the plus/minus of each, and who would want to be buying each one for what shooting style.
It's obvious he's got his personal favourite lenses (like the 70-200 2.8IS, frankly if i had his budget i'd probably never take it off the camera either).

For non-canon lenses it can be a bit of a mixed bag. It does seem sometimes he says a bit more often something like "i tried 4 copies of this <non canon> lens, every single one was different front/back focussing, one had bad centering, one the IS rattled and the other had a dead mouse between the elements". but more credit to him for trying out 4 and not giving up at the first bad one.
over at photozone.de, for some lenses that tdp says have bad QC, they say they loved it for all the QC reason tdp hated it. maybe they got one outlier? maybe tdp got 5 outliers? maybe there's a difference between the retail stuff they sell in the usa vs germany. all it says to me is to buy it in a shop and try it out, or online from a shop with a very good returns policy, and don't buy it the day before you need it.

tdp is sometimes harsher on 3rd party than canon. especially ziess, but if you've got zeiss money you expect good results so you have to be harsh. sometimes he says a 3rd party is not as good as the canon, but then says the canon is over twice the price and lets the reader decide if the next 5% in IQ is worth doubling the price. i've heard him be really harsh on canon lenses too, especially older kit lenses, moreso when there's a better 3rd-party for cheaper.

photozone put in the hard-yards too, MTF testing they try lots of bracket-focus shots, to negate field curvature, retesting if necessary, real numbers on CA and MTF i much prefer over tdp's sometimes-vague statements.



for cameras, neuro's already mentioned DxO mark for noise. if you're buying into a system from nothing then you also need to test the glass you have to buy. if you're already a canon-boy then there's nothing wrong with a tdp review. or even that, i'd just compare specs on canon.com and decide what features i need for my budget, there's not much choice in the top end, only in the xxxD budget range.


the one type of review that i do read are the user reviews from online retailers. but i also do drive a forklift full of salt and place it next to my laptop while i read. main point is that you never know the shooting style, subject, and quality-requirements of the reviewers, and you never know if they even work for whatever brand.


and whoever mentioned reading the negatives, me too. works for everything, i always went straight to the worst-rating picking accomodation at hostelworld when travelling, read through a few that just said 'it was crap' until i get to one that actually elaborated, then made a call as to whether it was the hostel, circumstances, or whether the reviewer was just a crybaby.

anyway, at the end, i'm with neuro. numbers don't lie, as long as you can trust the guy making the measurements and know the limitations that the numbers give...
Too much gear, too little space.
Gear Photos

Redreflex

  • Rebel SL1
  • ***
  • Posts: 90
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #11 on: July 14, 2011, 11:28:11 PM »
Just on TDP, it's a canon review site. he only reviews canon. any time he mentions nikon he says he doesn't own/use much nikon, if any.


Like Ken Rockwell and Nikons? Even though he also reviews canons...

neuroanatomist

  • CR GEEK
  • ********
  • Posts: 14047
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #12 on: July 15, 2011, 09:15:13 AM »
For a canon vs canon vs canon-fit-3rd-party, it's great. every review he compares each lens to a few others in the same price/size/zoom range category, and explains the plus/minus of each, and who would want to be buying each one for what shooting style.
It's obvious he's got his personal favourite lenses (like the 70-200 2.8IS, frankly if i had his budget i'd probably never take it off the camera either).

For non-canon lenses it can be a bit of a mixed bag. It does seem sometimes he says a bit more often something like "i tried 4 copies of this <non canon> lens, every single one was different front/back focussing, one had bad centering, one the IS rattled and the other had a dead mouse between the elements". but more credit to him for trying out 4 and not giving up at the first bad one.

Agreed, and I suspect this was Flake's point (i.e. Canon vs. 3rd party, rather than Canon vs. Nikon).  I did notice that in his review of his new favorite lens, the 70-200 f/2.8L IS II (which I agree is an incredible lens), he points out differences in the 3 copies that he tested (one has worse IQ at 200mm, better at 70mm, than the other two).  But he certainly doesn't make as big a deal of that as he does of copy variation in 3rd party lenses.
EOS 1D X, EOS M, and lots of lenses
______________________________
Flickr | TDP Profile/Gear List

canon rumors FORUM

Re: Biased Reviewing & the consequences.
« Reply #12 on: July 15, 2011, 09:15:13 AM »

Flake

  • Guest
Re: Biased Reviewing & the consequences.
« Reply #13 on: July 15, 2011, 09:34:14 AM »
Just to add to this from a slightly different angle.

I think all reviews of camera products should take account of the target market, and should take as it's starting point the people most likely to be buying it.  Purchasers of entry level cameras should be easily able to replicate the results a reviewer has obtained.  If it comes with a kit lens then it should be tested with that lens or a comparable one, there is little point in testing with a £2000+ pro grade lens on an entry level camera & suggesting that buyers will see similar results, nor taking shots on a tripod with mirror lock up because entry level users are unlikely to use such techniques.

Reviews need to fairly reflect what an average user can expect from a camera in their every day use of it, not what it is capable of when bleeding edge techniques or exotic glass is used.  blatant hyperbole, or worse lies help no one, least of all the manufacturer, when products don't live up to the hype.

motorhead

  • Canon 70D
  • ****
  • Posts: 267
    • View Profile
Re: Biased Reviewing & the consequences.
« Reply #14 on: July 15, 2011, 09:36:01 AM »
I take the view that any review is suspect and so are many magazine tests. Both have financial reasons for being as positive as possible and to play done the negatives. So I try to read as much as possible about a new camera or lens from as many different sources as possible.

However, its also true that expectations might to too high. Particularly in the area of noise. Yes, it is being reduced, but I get the impression it will never be completely iradicated and that it's a slow improvement.   

canon rumors FORUM

Re: Biased Reviewing & the consequences.
« Reply #14 on: July 15, 2011, 09:36:01 AM »