What about DxO Mark?

[Mt Spokane Photography]

Here's a great but totally meaningless comparison - EF 50mm f/1.8 II vs. EF 300mm f/2.8L IS II, both tested on a 5DII, a $100 lens vs. a $7000 lens.  The 50/1.8 II has a higher overall score (27 vs. 15) and better resolution (59 lp/mm vs. 49 lp/mm) - it must be better, right?   

Ha, I did not try that comparison.  It just confirms that their value rating point system is useless.  But then, most of us know that.  I much prefer those lens testers who add personal notes based on their actual use of the equipment along with their measurements.  Both are valuable, and one without the other is misleading.

I'd also like to see a database of lens focusing speed and accuracy.  I think its coming in the next year more or less.  Certainly, software like FoCal is providing a base for adding more and more lens parameters.  What is needed, of course, is a average of hundreds of tests, so we can begin to see if there is a particularly good or bad one with regard to focus accuracy.  Right now, its just individuals opinions or observations with one lens.

[ejenner]

After having some issues with my 17-40 and having it to be repaired and then sent back again, I started testing my lenses in real world situations with some scientific reasoning/experiments.  I was never much of a brick wall fan, so if it seemed to do what I wanted, I was OK - which is what got me into bother with the 17-40 after moving to FF.

But anyway, that's an aside, the one thing I really learned from all this is that each lens has a very definite set of characteristics that cannot be dumbed down into 1, 2 or even 3 numbers.  Or even the center, mid center and a corner on a (flat) test chart.

For instance, no test chart or DxO score tells me how quickly the resolution of the 17-40 drops off compared to say a 24-105 at 24mm f11.   (No, even a test chart does not).  In fact the more I actually look at real images, the less the lenses seem to be properly characterized buy the test scores even when I can see how those were obtained.

Point is modern lenses are complicated, they have a lot of glass and a lot of corrective elements that make things that should be simple (like DOF) more complicated.  Unless you are shooting wide open in the center or at f22 (no wait, not all lenses are the same at f22 despite the fact that every lens should be equally mushy from diffraction)........no, no not going there.

Blah. Rant over.   DxO, photozone, blah.  Maybe good for a quick general 'is this a lemon' glance, but even then I'm suspicious.  All those numbers for people to read so much into, reminds me of the difference between precision and accuracy.

[briansquibb]

I think DxO can measure the bodies reasonably well. It gives a broad brush view on the bodies capability - for example the comparison of the iso between the 7D and the 5DII gives the 5DII having better iso performance, however the actual numbers they give are pretty meaningless to the average user. The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.

Another example would be that the 1D4 is noiser than the 5DII at iso100 to iso200 and iit upsets some people. Personally I cant see the noise on an A3 print - so to me it isn't relevant like it is to others.

Another example was that on paper the 1ds3 should be trounced by the 5DII. In reality the 1Ds3 gives much better quality (to me) IQ  within its limited iso capability (native iso 1600 max) - again my opinion.

I dont look at DxO lens comparisons - I go to more real world review sites - such as Photozone ( http://photozone.de/reviews where they tell it as they see it rather than being reliant on pseudo scientific measurement assesment.

[JR]

The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.

Does anyone know if any body figured out a way to measure this "quality" of noise?  Like dpreview or other site/magazine?  I am curious to know because I have seen this comment a few time lately on this site and would like to better understand it!

Jacques

[nightbreath]

It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II

Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?

[ejenner]

The 'quality' of the noise is not mentioned and also the 'quality' is very much a perception that is different for each user.

Does anyone know if any body figured out a way to measure this "quality" of noise?  Like dpreview or other site/magazine?  I am curious to know because I have seen this comment a few time lately on this site and would like to better understand it!

Jacques

Not sure if I totally understand it, but say I have two images with the same signal/noise level, but in one the noise is high spatial frequency (speckly sp?) and the other is lower amplitude but also lower frequency.  The speckly noise will likely be less objectionable even if it is higher magnitude (for the same S/N it is likely to be).  It is less likely to be seen in print and more easily attenuated in NR algorithms.

This is once reason when comparing sensors I like to download raw files and process them.  Sometimes the slightly noisier raw file is actually easier to clean up in PP.   Having those raw files available is one of the things I really appreciate, although even then it is not perfect becasue a particular ISO on one camera is not necessarily the same exposure as that ISO on another camera.

In general the most objectionable noise is low-frequency chroma noise.  Small amount of luminance noise can actually make an image appear sharper.  Of course some of this is personal taste, possibly depending on what you compare it to.  I think people who used fast color film generally don't mind the speckle-look as much as the broad smeary-color look of a lot of chroma noise.

[ejenner]

Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?

I haven't looked at those specifically, but much of the improvements to date (the last few years) are indeed improved technology.  Actually much of it is reduced sensor read noise which basically comes down to being able to manufacture known electronics technology more cheaply.  This really helps with high ISO noise levels.

However, I would say that even the latest generations have some tricks, although 'cooking' raw files seems to be a new phenomenon.  I mentioned different ISO behavior, there may also be things about the smaller sensors that I am unaware of.

Honestly though, how does a crop sensor have a higher DR than a FF?  It would have to have a very much lower noise floor.  I'm not sure, but looking at DxO's definition it might well be to do with the way they compute their scores.  For instance it is possible that the 5DII will show more shadow detail, but with noise beyond the DxO cutoff, were as the smaller sensors suddenly become very noisy in the shadows.  Or maybe it is normalized so comparing different formats is not really applicable.  However, combined with the ISO scores, they don't really make sense to me.    Also you would improve the ISO score by processing the raw file, but even processing the raw file I find it hard to believe you would increase the DR that much.

EDIT:  I just looked at the S100 - DR 11.6 EV, almost as good as the 5DII?  Yea, that's normalized or something.

[unruled]

It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II

Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?

call in moore's law -- advancements in miniturization and the design of IC's is boosting improvement each year. If you take the 5D classic and compare its output at 1600ISO, and put it next to a 60D at 1600... full frame does not always win.

[well_dunno]

 70-200 f/2.8 IS II has a lower score than all the 70-200 lenses with the exception of f/4 non-IS version in DxO Mark lens rating list, which was disputed by some users there...

[elflord]

call in moore's law -- advancements in miniturization and the design of IC's is boosting improvement each year. If you take the 5D classic and compare its output at 1600ISO, and put it next to a 60D at 1600... full frame does not always win.

If you compare the DXOMark scores, you'll see that on the ISO component, it is more or less determined by sensor size. In the example above, the Nikon 7000 vs Sony NEX7 vs 5D Mark II, if you look at the details,

(1) the 5D Mark II wins on the ISO component, and is consistently better in the SNR18% chart.
(2) the other cameras do better on the dynamic range part of the DXOMark test due to better dynamic range at low ISOs. The 5D Mark II has better dynamic range at ISOs 800-3200. The NEX7 has a curious inflection point in its dynamic range chart. It looks as though  they change some in-body parameters at very high ISOs.

Regarding the 60D and the 5D classic, the 5D classic does beat the 60D on the ISO portion of the DXOMark, and at ISO 1600, it does better than the 60D on all measures in the DXOmark. In fact with the exception of dynamic range at ISOs under 400, the 5D classic beats the 60D on every measure in across its ISO range.

[unruled]

really? I hadn't checked dxo,but I've shot them side by side and felt the 60d did better. I don't know, maybe I have insufficient faith in artificial benchmarks.:-)

[elflord]

It's also interesting to know whether or not DxO sensor ratings is something we should be guided by. If you compare new APS-C sensor ranks even with current full-frame sensors, new ones will have better performance: DxO Mark Sensors Comparison: Sony NEX-7, Nikon D7000, Canon 5D Mark II

Can someone explain me what's happening? Is this an improved technology used by all manufacturers (Nikon, Sony, Pentax) or some kind of hack (algorithms applied to RAW files in-camera) manufacturers use to make their images look better than they initially are?

I think if you take a look at the graphs  (click the measurements tab and look at dynamic range and SNR18%), you can see where it comes from -- improved dynamic range at low ISOs. Maybe this suggests that sensor dynamic range is improving, but signal to noise is largely constrained by the sensor.The numbers are consistent with this reading of the graph -- the dynamic range score is based on the low ISO part of the dynamic range curve. The ISO component of the benchmark reflects performance in the higher ISO range on SNR, dynamic range and bit depth curves.

[LetTheRightLensIn]

I think the most of you know DxO Mark and there Sensor Benchmarks.

They also test lense and i recently seen that most of Tamrons SP Lenses are mostly at the same quality as the L-Senses of Canon.

But they cost only 1/4 of the L-Lense of Canon.

They have a proper test which is not to misunderstand and tell very exactly how they test, also the results seems to be very realistic.

My Question is why? Why are the Tamron SP Lenses at the same quality as the L-Lenses of Canon, but cost only 1/4.

I doubted it and bought some L-Lenses and SP Lenses with the same focal length for example 70-300mm.

really, the SP lense from Tamron seems to own the Canon lense in nearly every way but cost only  ~330 €.

The Canon L-Lense 70-300 is about ~1250 €

I tested a lot of things, outside an inside. Checked the lp/mm with a lot of test photos and it seems to be true.

At DxO even the before version of the current 70-300 without IS and without SP seems to be a hard competitor for the L-Lense.

With the lenses, it seems that those L-Lenses from Canon are something like a religion. You just believe that its the best on the marked and so it worths the money.

But Tamron is no replace for Canons EF-Lenses, Tamron have only a hand-full ob objectives and only 30-50% of them (depends on APS-C or Full-Frame) are SP-Lenses.

Or do you think DxO is wrong at all?

I think they have good sensor data but terrible lens data.

I did a super careful controlled comparison between the Tamon 70-300 VC and Canon 70-300L and my results did not remotely match their results. Nor have the results from most other lens test sites.

They have weird stuff, they say that at 200mm f/2.8 that the 70-200 f/2.8 IS is sharper than the 70-200 2/8 non-IS is sharper than the 70-200 2.8 IS II. Pretty much everyone who has used them has found the 100% exact opposite results.

They claim that the wide angle Canon FF zooms are sharper at the edges at f/2.8 than at f/5.6. Very bizarre.

I believe they ranked the Canon 70-300 IS non-L better than the Canon 70-300L at 300mm. All I can say is that sure didn't match my results. In fact I think they even had the 70-300L coming in second to last for 300mm and the 300mm IS coming in last! Something crazy like that.

Lens testing is VERY tricky. I bet they just do one quick AF or MF try and are done with it when and that would lead to very random results.

Anyway I do not think the Tamron matches the 70-300L, at all in center frame sharpness anywhere near wide open anywhere along the entire focal range (it gets closer at the edges, oddly enough, or when stopped to f/8, not so surprising). The Tamron AF is also only half the speed. But the price is amazing so it's certainly the best value in tele-zooms.

And I actually do think their 17-50 2.8 is better overall, on APS_C only of course, than the Canon 17-40L, even though it costs less. I added the Tamron, compared them and sold off the L. They don't pay as much for marketing and can charge less than someone like Canon.