I tested out the theory behind DxOMark's F-Stop blues article

[Kit Lens Jockey]

I was kind of put off when I read the DxOMark article referencing how very fast lenses on digital cameras are a waste due to the sensor's limited ability to gather light from a lens with a very wide aperture. You can read the article here, but the basic gist is that digital sensors are somewhat incapable of registering light from very wide aperture lenses, so cameras secretly bump up the ISO at wide apertures in order to compensate for this.

https://www.dxomark.com/Reviews/F-stop-blues

I really didn't want to believe that dropping serious money on a 50mm f1.0 was completely pointless, so I decided to do a sort of "blind" test with my 5D Mark III.

I set the camera up on a tripod, attached a Canon 50mm 1.4 lens to the camera, obtained focus, and then removed the lens, stuck a piece of tape over the electrical contacts on the lens so that the camera could not discern what kind of lens was attached to it, and I took a photo in manual mode at 1/50 of a second and ISO 1000. I then did the same thing with a Canon 50mm 1.0 lens.

This gave me two photos, one taken with a 1.4 lens wide open, and another with a 1.0 lens wide open, both at the same camera settings, and both taken so that the camera had no way of knowing what lens it was taking the photo with.

The result? the 50mm 1.0 was noticeably brighter than the 50mm 1.4. I guess if I really wanted to I could get into a big technical analysis of whether or not the 1.0 lens gave exactly a 1 stop increase in brightness or not. But for me, right now, just knowing that having a 1.0 aperture does actually make a difference on a digital camera is enough.

Here are the photos. Photo 5540 is with the 50mm 1.0 lens, 5541 is with the 50mm 1.4. The only modifications I made to them were to convert them both to sRGB and resize them down to 3000px. I admit the focus on the 1.4 image is pretty bad, but for the purposes of this, I wasn't too concerned with perfect focus.

 

[raptor3x]

Playing around with the exposure of the two images in LR and looking at the histograms I'd say there's only ~1/3-1/2 stop difference in exposure between the two, which is what you would expect if you extrapolated DxO's data.

 

[Kit Lens Jockey]

Hmm, interesting. Maybe it is pointless to have this lens after all. Is going to f1.2 from f1.4 more significant of a difference than going from f1.2 to f1.0? The wide aperture 50mm that I most commonly use is the Sigma 50mm art, but I was considering changing to the Canon 50 1.2 to gain that extra little bit. The f1.0 is fun to play around with, but I'm too worried about something happening to such an expensive and non-repairable lens to it to use it frequently.

 

[Kit Lens Jockey]

So, I was curious to see how the new sensor in the 5D4 would handle super wide apertures in comparison to the 5D3. So, I got out the 50mm f1.0, focused it, taped off the electrical contacts, and mounted it on a 5D3 and then a 5D4 back to back at the same shutter speed and ISO settings.

Both of these are shot at a 10 second exposure and an ISO of 100. The cameras they were taken with are in the file names. These are straight out of the cameras, just re-sized to the same size and converted to sRGB. Unfortunately I moved the tripod a little while switching cameras. It might just be the difference in white balance, but the 5D3 actually looks brighter than the 5D4. :-\

Could this mean that the 5D4's higher megapixel count hinders its ability to pull in light from very wide aperture lenses? Hard to say, but seems possible.

 

[AlanF]

The real iso sensitivity often differs from the nominal, as measured by DxO. The MkIII is reported to be slightly higher iso than the MkIV. https://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-EOS-5D-Mark-IV-versus-Canon-EOS-5D-Mark-III___1106_795

 

[C-A430]

What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4

24-105 f4 never gets more than t5 - it is partly because aperture is not open at f4 and partly because its low quality glass reflects part of the light.

 

[Kit Lens Jockey]

What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4

24-105 f4 never gets more than t5 - it is partly because aperture is not open at f4 and partly because its low quality glass reflects part of the light.

And how exactly was I supposed to stop down the f1.0 lens to f1.4 while also preventing it from communicating with the camera so that the camera could not make its own ISO adjustments behind the scenes as the article states?

 

[neuroanatomist]

What is the point of comparing between different lenses? You were supposed to test the same lens at f1.0 vs f1.4

24-105 f4 never gets more than t5 - it is partly because aperture is not open at f4 and partly because its low quality glass reflects part of the light.

And how exactly was I supposed to stop down the f1.0 lens to f1.4 while also preventing it from communicating with the camera so that the camera could not make its own ISO adjustments behind the scenes as the article states?

Set the aperture to f/1.4 (or whatever), press and hold the DoF preview button, then unlock and very slightly rotate the lens barrel so the electronic connection is broken.

 

[Kit Lens Jockey]

Aren't the apertures spring loaded to return to full open when power is removed from the lens? Or does it require power to move each way? Either way, I don't want to fool around with such a rare lens too much.

 

[neuroanatomist]

Aren't the apertures spring loaded to return to full open when power is removed from the lens? Or does it require power to move each way? Either way, I don't want to fool around with such a rare lens too much.

Well, I don't make a habit of suggesting things that don't work. Incidentally, that 'trick' is how people who use macro reversing rings are able to use a stopped-down lens. It won't harm the lens.

As was pointed out, using two different lenses, which have different T-stops relative to their F-stops, is a significant confounding factor in your test.

 

[midluk]

You could shoot the same f/1.0 lens at f/1.0 with and without the contacts taped to see how much correction the camera applies.

By having a look at the aperture scaling values in the camconst.json file of RawTherapee, you can learn that it indeed seems like the 5D4 has a higher scaling than the 5D3. There are no measured values below f/2 yet for the 5D4, though.
So to get real results, shoot a completely saturated image with the f/1.0 lens with different apertures and see how the camera scales the saturation values in the raw file. Assuming Canon has implemented correct scaling factors in the camera, that should be the numbers you are searching for.

 

[gruhl28]

Interesting subject, Kit Lens Jockey; thanks for taking a first stab at testing this. I agree with some of the previous comments, though - you can't draw conclusions comparing different lenses as they may have different transmission. And it would have been better to adjust the shutter speed also when comparing f/1.0 with f/1.4.

I think midluk had the best suggestion - shoot at f/1.0 with the contacts taped and untaped and compare - that will show how much the camera compensates. You could also shoot at f/1.4 using Neuro's procedure, but halve the shutter speed (twice as long). The exposure at f/1.0 at one shutter speed and the exposure at f/1.4 at half the shutter speed (twice as long a shutter speed) should look the same if the sensor is able to capture all the light. If the f/1.0 exposure is darker, then you have proven what DxOMark has written. You can then use Lightroom or PS to see how much darker the f/1.0 exposure is.

Please let us know the results.

 

[IglooEater]

You could shoot the same f/1.0 lens at f/1.0 with and without the contacts taped to see how much correction the camera applies.

+1.

Op, thanks for doing the tests, we don't all have a 1.0 to experiment on!

 

[hendrik-sg]

For me, the comparision is reasonable between 2 lenses. Maybe it depends on what for one buys a fast lens.

The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.

For me this is a big point, because i use the fastlenses for low light applications. In the 35m range it would be specially interesting, just because the 2.0 IS lens is really good and much less expensive then the 1.4 version and has IS on top

 

[Mikehit]

For me, the comparision is reasonable between 2 lenses. Maybe it depends on what for one buys a fast lens.

The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.

For me this is a big point, because i use the fastlenses for low light applications. In the 35m range it would be specially interesting, just because the 2.0 IS lens is really good and much less expensive then the 1.4 version and has IS on top

It depends on what the question is.
As I understood the OP, at very wide apertures the camrea bumps up the ISO artificially but at narrower apertures with the same lens it does not (need to?) do that.
So if you use two different lenses, you have to consider not only the amount of light coming into the camera through the aperture but also the amount of light the glass itself transmits. If the two lenses transmit different amounts of light then the comparison becomes flawed especially if the false-ISO is accounting for less than one stop of illumination.

Then again my assumption behind the OP may be incorrect.

 

[neuroanatomist]

I really didn't want to believe that dropping serious money on a 50mm f1.0 was completely pointless, so I decided to do a sort of "blind" test with my 5D Mark III.

But for me, right now, just knowing that having a 1.0 aperture does actually make a difference on a digital camera is enough.

The testshows, that the 1.0 lens transmits not much more light than the 1.4 lens. To gatter most light possible, the 1.0 is of very few value, it seems to be only about bokeh then.

Interesting that two people look at the same images and come to opposite conclusions... The one who spent $$$$ on a 50/1.0L concludes that it was money well spent, the one who didn't spend $$$$ concludes it would not be worth spending it.

'Round these parts, we call that bias.

 

[scottburgess]

The conclusion made in the article is that this is a problem with the sensors. However, the data presented do not fully support this conclusion. To more carefully tease out the relative contributions of lens and sensor one could do this experiment: hook up a Canon 35mm film camera, load some Fuji Velvia, and compare exposures of a flat target at f/1.2-1.4 with the same exposure at other apertures.

If the exposures are noticeably different, then the lens could be contributing to the problem. If the film frames are indistinguishable (ignoring DOF and vignetting), then the sensor and gain are probably the primary culprits. I am wondering if vignetting or metering matters here, or if the manufacturers are cheating the measurement of the maximum aperture by 10-20% or so like they do for the focal length of lenses.

Anyone want to try that out? Might be fun if the article turns out to be wrong. I'm not funding the research, mind you, but will kibitz from the sidelines.

 

[Mt Spokane Photography]

I've always thought that the reason for buying a ultra wide aperture lens was more related to shallow depth of field than getting more light. There are known ways to design cameras and lenses to do a better job at ultra wide apertures, but the cost is very high. Still, we have seen recent patents trying to improve on the situation by using modified pixels or microlenses near the edges so they pick up more light.

 

[sulla]

IF the sensor really didn't at all register light incident at high angles (coming form the outermost part of the lens, i.e. from the parts of the lens between 1.4 and 1.0) then the bokeh should not be different either. So, if this was true, very high aperture lenses should not provide better bokeh than more narrow lenses.
So an alternative test for the DxO-claimed effect could be to compare the diameter of bokeh circles of the EF50 1.0 at 1.0 and at 1.4. In this test it would not be necessary to block the contacts, as the camera could forge ISO, but cannot forge bokeh.

On the other hand, IF the lens transmission in the outermost regons was only gradually reduced, this would make all and every very high aperture lens an apodization lens, making its bokeh only much smoother. So, I guess, the money would be well spent...
A test could be done to compare the shape of bokeh circles of the EF50 1.0 at 1.0 and 1.4. If circles at 1.4 show a distinctive edge and those at 1.0 don't, then this should measure the "sensor apodization".

 

[applecider]

Doesn't vignetting difference at progressively wider apertures account for the lower amounts of light that DXO is measuring? And shouldn't the actual iso be noted in some exif if it is being changed under the hood? And finally the newest body that I saw listed in the referenced article was the 5d, just seemed odd. Granted it was 2010 would current bodies handle iso in the same way one would want to know. That is change to a higher effective iso as apertures get larger, and were in body lens corrections on and available then?

And to the original poster, where did you get those models? Voodoo? They kinda creeped me out.