Biased Reviewing & the consequences.

[Flake]

Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading. This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.

For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.

 

[UncleFester]

Are you talking about the beta-testers that furnish a review?

Or someone who just bought one and wrote a review.

When I look for reviews I'm always looking for the most negative. They are usually more realistic.

 

[Canihaspicture]

I used a P7000 on a photo walk and it was garbage with a battery. However, many of the people on the same walk loved it. I'm not sure why, maybe they weren't in manual mode. It could be like an Apple brainwash thing like in the Cult of Steve Jobs, but only for Nikon.

The camera couldn't accomplish basic camera functions without problems. I'm not even mentioning the speed of the thing yet.

Personally, I like DigitalRev on youtube for reviews and commentary. That dude tells it how it is and he's funny. He is an admitted Nikon guy and even he says the P7000 sucks.

 

[Flake]

No not beta testers, the so called more respectable ones - magazines and certain websites. For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I just think it's so unfair of reviewers to hype a product, and then have to read people wondering why they can't get the results the reviewer has claimed.

 

[Mt Spokane Photography]

Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading. This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.

For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.

Anyone who does not agree with me is biased!

That is just human nature. We often see people talking about using real world images to test a camera, which, in fact, can be unintentionally selected to hide defects or make them less apparent. There are often no established tests for cameras, so opinions are all we have. For example, we hear that AF is fast or it is slow with a lens. How do you measure it? Its a function of lens, body, subject lighting and contrast, and probably more.

As someone who has had to conduct lab testing for use by some very technical and particular customers, I eventually learned to separate facts and opinions in my conclusions, and to justify my opinions. I have had to hold back a lot of opinions because there was no hard evidence to justify it.

We all see things thru a filter which is based on our experiences. If I've had a issue with a product in the past, I tend to be unforgiving when it comes to reviewing it. for example, I have had four sigma lenses that were made to be EF compatible, but did not work on DSLR's because Sigma did not understand the Canon lens communication adequately. Even now, I avoid Sigma, and when I see comments about poor autofocus accuracy, it just reinforces my experiences.

 

[UncleFester]

No not beta testers, the so called more respectable ones - magazines and certain websites. For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I just think it's so unfair of reviewers to hype a product, and then have to read people wondering why they can't get the results the reviewer has claimed.

I don't think a lot of reviews - magazine, consumer, some websites - are as thorough as, say, DPreview. Meaning there are probably more "knee-jerk" reviews with very little effort, given the images some present as evidence of their research - for example, leaning out the office window and shooting the buildings across the street. hardly realistic examples.

Same goes for retail reviewers.

 

[steven63]

DPreview all the way.

 

[Flake]

Before I start this thread I'll make it clear that this is a swipe at reviewers who have allowed bias to colour reviews, and not at camera manufacturers in any way.

The Nikon D7000 has received some glowing reviews to the point that some have stated that it's almost as good on noise as the D3 and yet I'm hearing so many stories from people who have bought this camera, that in practice it does not live up to the hype.

I don't believe that this is a problem with the camera, but with reviewers who have so over egged the pudding, that the reviews they have written have been downright missleading. This is helpful to no one, especially Nikon when buyers are dissapointed, nor to buyers wondering why they can not achieve the kind of results which the reviews are claiming.


For a long time I've been aware that a few dinosaurs who were comitted to Nikon in the days of film have been unable to accept the change to digital, and retain such an affection for one brand that they are unable to produce an impartial, and uncoloured review based on fact.

Anyone who does not agree with me is biased!

That is just human nature. We often see people talking about using real world images to test a camera, which, in fact, can be unintentionally selected to hide defects or make them less apparent. There are often no established tests for cameras, so opinions are all we have. For example, we hear that AF is fast or it is slow with a lens. How do you measure it? Its a function of lens, body, subject lighting and contrast, and probably more.

As someone who has had to conduct lab testing for use by some very technical and particular customers, I eventually learned to separate facts and opinions in my conclusions, and to justify my opinions. I have had to hold back a lot of opinions because there was no hard evidence to justify it.

We all see things thru a filter which is based on our experiences. If I've had a issue with a product in the past, I tend to be unforgiving when it comes to reviewing it. for example, I have had four sigma lenses that were made to be EF compatible, but did not work on DSLR's because Sigma did not understand the Canon lens communication adequately. Even now, I avoid Sigma, and when I see comments about poor autofocus accuracy, it just reinforces my experiences.

I'd agree with all of that, but one of the problems has been with noise, where some reviewers have claimed that the D7000 is as good as the D3 when it clearly is nothing like as good. Opinion is one thing, however exaggeration is something else, and is misleading. One of the complaints has been from a user trying to replicate low light results & then wondering what he's doing wrong when it's nothing like as good as the review had led him to believe.

If it was just a few reviewers opinions I'd never have posted the thread, however this is blatent over hyping a product which is adversly affecting the product experience of users, dissapointing them greatly.

 

[Loswr]

I'd agree with all of that, but one of the problems has been with noise, where some reviewers have claimed that the D7000 is as good as the D3 when it clearly is nothing like as good. Opinion is one thing, however exaggeration is something else, and is misleading.

As a scientist, I like numbers. They don't glow, they don't hype. DxOMark reports the low-light sensitivity, defined as the highest ISO sensitivity to which your camera can be set while maintaining a high quality, low-noise image (based on a Signal-to-Noise-Ratio [SNR] of 30dB, a dynamic range of 9EVs and a color depth of 18bits), as ISO 1167 for the D7000 and ISO 2290 for the D3, so the D3 over a stop better in low light.

No not beta testers, the so called more respectable ones - magazines and certain websites. For cameras I do tend to believe DPreview, and in terms of Canon products take digital picture with a pinch of salt because it's also biased.

I'm curious as to how Digital Picture is biased, in your opinion? I do see that the reviews there are 'rose-colored' in the sense that the author never comes right out and says something like, "This lens is a cheap piece of crap," and he finds something good to say about almost every lens (even if the IQ is low, the lens is great for a budget, etc.). But I wouldn't call that a bias, more like a viewpoint.

 

[Flake]

I think digital picture is biased towards Canon products to the point where an unduely positive gloss is placed on some products and the third party brands suffer the opposite effect. All this is fine if you know about it, but it could put someone off buying one lens if they weren't aware.

 

[dr croubie]

Just on TDP, it's a canon review site. he only reviews canon. any time he mentions nikon he says he doesn't own/use much nikon, if any.

For a canon vs canon vs canon-fit-3rd-party, it's great. every review he compares each lens to a few others in the same price/size/zoom range category, and explains the plus/minus of each, and who would want to be buying each one for what shooting style.
It's obvious he's got his personal favourite lenses (like the 70-200 2.8IS, frankly if i had his budget i'd probably never take it off the camera either).

For non-canon lenses it can be a bit of a mixed bag. It does seem sometimes he says a bit more often something like "i tried 4 copies of this <non canon> lens, every single one was different front/back focussing, one had bad centering, one the IS rattled and the other had a dead mouse between the elements". but more credit to him for trying out 4 and not giving up at the first bad one.
over at photozone.de, for some lenses that tdp says have bad QC, they say they loved it for all the QC reason tdp hated it. maybe they got one outlier? maybe tdp got 5 outliers? maybe there's a difference between the retail stuff they sell in the usa vs germany. all it says to me is to buy it in a shop and try it out, or online from a shop with a very good returns policy, and don't buy it the day before you need it.

tdp is sometimes harsher on 3rd party than canon. especially ziess, but if you've got zeiss money you expect good results so you have to be harsh. sometimes he says a 3rd party is not as good as the canon, but then says the canon is over twice the price and lets the reader decide if the next 5% in IQ is worth doubling the price. i've heard him be really harsh on canon lenses too, especially older kit lenses, moreso when there's a better 3rd-party for cheaper.

photozone put in the hard-yards too, MTF testing they try lots of bracket-focus shots, to negate field curvature, retesting if necessary, real numbers on CA and MTF i much prefer over tdp's sometimes-vague statements.



for cameras, neuro's already mentioned DxO mark for noise. if you're buying into a system from nothing then you also need to test the glass you have to buy. if you're already a canon-boy then there's nothing wrong with a tdp review. or even that, i'd just compare specs on canon.com and decide what features i need for my budget, there's not much choice in the top end, only in the xxxD budget range.


the one type of review that i do read are the user reviews from online retailers. but i also do drive a forklift full of salt and place it next to my laptop while i read. main point is that you never know the shooting style, subject, and quality-requirements of the reviewers, and you never know if they even work for whatever brand.


and whoever mentioned reading the negatives, me too. works for everything, i always went straight to the worst-rating picking accomodation at hostelworld when travelling, read through a few that just said 'it was crap' until i get to one that actually elaborated, then made a call as to whether it was the hostel, circumstances, or whether the reviewer was just a crybaby.

anyway, at the end, i'm with neuro. numbers don't lie, as long as you can trust the guy making the measurements and know the limitations that the numbers give...

 

[Redreflex]

Just on TDP, it's a canon review site. he only reviews canon. any time he mentions nikon he says he doesn't own/use much nikon, if any.

Like Ken Rockwell and Nikons? Even though he also reviews canons...

 

[Loswr]

For a canon vs canon vs canon-fit-3rd-party, it's great. every review he compares each lens to a few others in the same price/size/zoom range category, and explains the plus/minus of each, and who would want to be buying each one for what shooting style.
It's obvious he's got his personal favourite lenses (like the 70-200 2.8IS, frankly if i had his budget i'd probably never take it off the camera either).

For non-canon lenses it can be a bit of a mixed bag. It does seem sometimes he says a bit more often something like "i tried 4 copies of this <non canon> lens, every single one was different front/back focussing, one had bad centering, one the IS rattled and the other had a dead mouse between the elements". but more credit to him for trying out 4 and not giving up at the first bad one.

Agreed, and I suspect this was Flake's point (i.e. Canon vs. 3rd party, rather than Canon vs. Nikon). I did notice that in his review of his new favorite lens, the 70-200 f/2.8L IS II (which I agree is an incredible lens), he points out differences in the 3 copies that he tested (one has worse IQ at 200mm, better at 70mm, than the other two). But he certainly doesn't make as big a deal of that as he does of copy variation in 3rd party lenses.

 

[Flake]

Just to add to this from a slightly different angle.

I think all reviews of camera products should take account of the target market, and should take as it's starting point the people most likely to be buying it. Purchasers of entry level cameras should be easily able to replicate the results a reviewer has obtained. If it comes with a kit lens then it should be tested with that lens or a comparable one, there is little point in testing with a £2000+ pro grade lens on an entry level camera & suggesting that buyers will see similar results, nor taking shots on a tripod with mirror lock up because entry level users are unlikely to use such techniques.

Reviews need to fairly reflect what an average user can expect from a camera in their every day use of it, not what it is capable of when bleeding edge techniques or exotic glass is used. blatant hyperbole, or worse lies help no one, least of all the manufacturer, when products don't live up to the hype.

 

[motorhead]

I take the view that any review is suspect and so are many magazine tests. Both have financial reasons for being as positive as possible and to play done the negatives. So I try to read as much as possible about a new camera or lens from as many different sources as possible.

However, its also true that expectations might to too high. Particularly in the area of noise. Yes, it is being reduced, but I get the impression it will never be completely iradicated and that it's a slow improvement.

 

[AJ]

The D7000 is arguably the best crop camera on the market today, sensor wise. It should be receiving glowing reviews. It's an awesome camera. I wonder what those folks who were disappointed with this camera were expecting.

 

[Mt Spokane Photography]

The D7000 is arguably the best crop camera on the market today, sensor wise. It should be receiving glowing reviews. It's an awesome camera. I wonder what those folks who were disappointed with this camera were expecting.

I love to see photos, post some. I won't criticize them, but its always nice to have photos from someone who praises or criticizes a camera.

 

[Flake]

The D7000 is arguably the best crop camera on the market today, sensor wise. It should be receiving glowing reviews. It's an awesome camera. I wonder what those folks who were disappointed with this camera were expecting.

One of the dissapointed was a D3 user who had taken the D7000 out in low light because a reviewer had said that it was as good as the D3 in low light. It isn't, and it's unfair to both the buyer and the manufacturer to raise expectations to false levels.

 

[Mt Spokane Photography]

The D7000 is arguably the best crop camera on the market today, sensor wise. It should be receiving glowing reviews. It's an awesome camera. I wonder what those folks who were disappointed with this camera were expecting.

One of the dissapointed was a D3 user who had taken the D7000 out in low light because a reviewer had said that it was as good as the D3 in low light. It isn't, and it's unfair to both the buyer and the manufacturer to raise expectations to false levels.

Undoubtedly true, but first hand reports are a lot more convincing than a report saying that I once heard a guy say ...

 

[Edwin Herdman]

I just skimmed through some of the comments. Time to correct the common views of The Digital Picture:

But first, of course everybody needs to use their best judgement when reading a site. I agree with neuroanatomist that numbers are helpful, although they need to be taken in context. Some cameras that rank highly on DXOMark are not appropriate for some uses compared with others, and of course money is a part of that equation.

Specific criticisms of Bryan Carnathan's site I spotted here:

"He only reviews Canon." This isn't correct. He's uploaded a lot of images (which I gather he took himself) of Nikon F-mount lenses, and notes he plans to review a lot of them. He started out with Canon, but he has reviewed Sigma lenses (his comments on the 120-300mm f/2.8

"He never comes right out and says something is junk." Maybe he's a bit more diplomatic than some, but he definitely gives a bottom line for each piece of equipment. Sometimes he says a lens is one of his favorite, most-used, or most recommended (from memory, the 70-200mm series covers the first two, and the 17-55mm and the 100-400mm L are examples of the last two), but sometimes he also plainly says when a lens is cheap and doesn't surpass its price segment in terms of quality and even that it should be avoided for an alternative. For the Canon 50mm f/1.2, for example, he says that it isn't sharper than the f/1.4 at some apertures, and the price is hard to justify unless you need its qualities. Seems like a fair assessment to me. I also note that he went to pains to do a side-by-side comparison of the defocus area quality (boke) of all three lenses in that focal length because they are commonly compared. Can't ask for much more.

The resolution charts are amazingly complete and comprehensive, I'm not aware of anything like it elsewhere on the web. (Although I wonder how many lenses look better or worse than their usual uses due to apparently being shot at different distances for the sake of a consistent magnification; some lenses seem to perform much better in close focus - that's been my observation, anyway).

Ken Rockwell (avoiding the temptation of a direct comparison) may occasionally put in "Photoshop lens correction factors," so there are useful resources out there if you know where to look. You might have to go through some text to get what you need with no guarantees, on either site, but all the information is generally helpful. Ken Rockwell is big on talking about compatibility with film lenses (in this day and age!) and Bryan Carnathan's reviews tend to be a complete tour of a lens (from his perspective but he covers things that should be useful for most everybody). Both guys could do more, but I don't pay them a subscription to put out content that I personally need, so I see no need to complain. When I want more information about a lens (i.e. is it parfocal and/or does it suffer focus breathing) I know to look other places as well (though sometimes you can infer that from the text).

My own personal (small) gripes are that Bryan sometimes does not really demonstrate the possibilities of a lens (mainly with TS-E lenses where he really could grab the bully pulpit and show off some of their great effects which people generally need more help grasping) and sometimes the image samples are hard to judge and not representative of all uses of a lens. But, like I said, this is not a big problem because other resources on the Internet cover those angles.

Photozone.de is another useful site, but sometimes you have to just look at the data and do your best to compare across formats (you can make a guess if there is a lens tested on both that you are using as a central comparison point) where there are holes in the data, or when the stated conclusion seems to be flatly contradicted by the actual test (the recent review of the Sigma 120-300mm f/2.8 APO EX DG OS is an example of this - given between three to almost four stars in optical quality on full format just because the edges of the frame get marginally softer than the center; all the samples shown pretty much completely contradict this finding, and some are outstanding and couldn't be expected from any other lens on this format, so yes, Photozone falls over too sometimes).

The only way to make a stab at overcoming bias is to read, read, read and view EVERYTHING you can find on the internet (and if possible investigate in person and talk to users in person) about a given piece of equipment.