Well, IMO it's actually worse than this. Because they would publish overall scores, but those scores are only reflective of the engine performance. There is no consideration for handling, traction, braking, interior features, etc.
If they tried to incorporate all of those features into their score, it would make things worse (and make the score more open to criticism).
It's better that they stick with a single thing -- benchmarking sensor performance.QuoteI don't know anyone who would buy a car based on engine performance charts, yet somehow this makes sense in a camera?
This is a straw man. No one has proposed that sensor performance is the only factor a buyer should consider.
You seem to be suggesting that no-one should measure engine performance (and therefore no-one should publish engine power/torque output and curves)
Not saying no one should measure, but that in the camera world these measures are blown way out of proportion for what they should be. And if they're truly just measuring sensors, then shouldn't be the rating be identified to the sensor, and not the camera?? Pretty sure multiple camera's use the same sensor, but you don't find the sensor score, you find the camera score.
My gripe is not in their testing, it's in the presentation and marketing layer. For example, they say "The Overall Sensor Score is based on all characteristics of a camera sensor, independent of the camera lens..." Umm, it's independent of a whole lot more than just the lens. My point is there is a certain amount of deceptiveness in their marketing (for lack of a better word.) They present scores in "Portrait" and "Landscape" and "Sports" even though their criteria do not apply to many critical elements of those photographic activities. Yes, they explain what they mean by those terms, but to the average consumer the name is going to mean a whole lot more than the actual test criteria. I know for my sports photography, the sensor itself is the least of my concerns and yet they present low-light ISO as a singular element for their "sports" score. Again, not an outright lie, but it sure seems intentional to mislead people that aren't willing or interested in reading the fine print.