FoCal Database for Lens Quality of Focus

[bluenoser1993]

I have found it makes a difference in my testing for just one case only, 400mm DO II + 2xTC + 5DIV, where RAW has much higher QoF than for jpeg. It makes negligible differences for all other combinations I have used whether RAW or jpeg is used, but jpeg is much faster.

Thanks Alan, I've done RAW in the past, but not with the 5Ds. I won't bother with the extra time.

 

[FoCal Rich]

In all "normal" test cases, it's fine to use JPEG analysis (and it's certainly a lot quicker).

Some info about raw analysis:

For raw processing we use a custom demosaicing algorithm which keeps all three colour channels totally isolated (unlike a processed JPEG which will have bleed of information between the colour channels). If you're trying to look at something very specific to individual colour channels or you're analysing under a different light source to normal (e.g. monochromatic light) then you'll want to use raw.

Also, using raw will ensure any camera processing effects are not applied to the analysed image - e.g. vignetting correction, distortion correction etc, white balance colour shifts etc. However, FoCal does adjust picture style, white balance and checks for various settings which may affect analysis before running tests so again in most cases you won't worry about this.

There is new processing related to Dual Pixel Raw in FoCal specifically for the 5Dmk4 - we use the dual pixel information to increase confidence in the AF Microadjustment result, to show focus offset in the AF Consistency results and to give an idea of lens focus shift in the aperture sharpness test. This blog post (http://www.reikan.co.uk/focalweb/index.php/2016/09/bringing-dual-pixel-raw-to-reikan-focal/) give more detail about this.


Just to re-iterate - you almost certainly don't need to use raw processing mode. Unless you understand specifically why you need raw, then you don't need to use it. It won't degrade the results you get to use raw unnecessarily, but it just takes longer!

Hope this helps.

 

[AlanF]

Rich
Can you think of a reason why I get higher QoF with RAW than jpeg for just one isolated case of a particular lens combination, the 400mm DO II + 2xTC + 5DIV? It's a particularly bad example with FoCal in general; the profile of QoF against AFMA is very flat, maybe because of the f/8 aperture and long distance to target, and the r, g and b channels some times give widely different optimal AFMA.

I don't get the same QoF discrepancy on a 5DS R with the same lens.

 

[FoCal Rich]

Rich
Can you think of a reason why I get higher QoF with RAW than jpeg for just one isolated case of a particular lens combination, the 400mm DO II + 2xTC + 5DIV?

Actually, yes there is a possible reason. We use a third party tool for decoding and some basic preprocessing of raw files, and this utility hasn't yet been updated to support the 5Dmk4.

FoCal 2.3 had an significant issue with contrast on 5Dmk4 raw files (i.e. the resulting analysed image could sometime have wildly incorrect contrast), but this was mostly corrected in FoCal 2.4. However, there may still be some issues under certain conditions so I'd be more inclined to trust JPEG results from the 5D4 if you get wildly differing values until this is fully fixed. (Note that this is JUST for the 5D4 - no other cameras)

 

[AlanF]

Discovered a flaw in the comparisons. For telephotos, the spread given by FoCal represents the entire focal length range and not the individual focal length you are measuring. For example, for the Canon 100-400mm it will give the overall spread for measurements at 100mm and for 400mm, which reads the same when you compare yours at 100mm and at 400mm. So, for a lens like the Sigma 150-600mm, which is very good at 150mm and not as good at 600mm, it gives the overall spread of both and you think that your 150mm end is better than average and your 600mm is worse!

 

[bluenoser1993]

Discovered a flaw in the comparisons. For telephotos, the spread given by FoCal represents the entire focal length range and not the individual focal length you are measuring. For example, for the Canon 100-400mm it will give the overall spread for measurements at 100mm and for 400mm, which reads the same when you compare yours at 100mm and at 400mm. So, for a lens like the Sigma 150-600mm, which is very good at 150mm and not as good at 600mm, it gives the overall spread of both and you think that your 150mm end is better than average and your 600mm is worse!

Interesting. So this makes my 100-400 II a little better when compared to the 5DsR numbers. At 400 I was only getting 1553, pretty well bottom of the average, but at 100 the QoF is 1683 which is at least a little more like middle of the road. Funny they would report it that way, it would be simple to have separate results with all the data they are collecting.

 

[FoCal Rich]

Actually, at the moment the data is split into 3 focal length regions (widest third, mid third and telephoto third). If there's enough data to pass the quality threshold (for the specific camera and lend), then the data for the specific focal length region is used for comparison, otherwise the combined data is used to give an idea.

After the next release of FoCal, we will we will be working a lot more with the comparison data so comparison at focal lengths may be made more specific.

 

[AlanF]

For the 2 I have looked at, the Sigma 150-600mm is definitely an average value, and the. 100-400mm II probably is (both ends are the same, but the lens is fairly constant along its length). The extender data look pooled.

 

[FoCal Rich]

I've had a quick play and generated the following 2 charts. These show the range of median data for all the camera bodies combined together for the 4 possible "focal length ranges" (wide, mid, tele and combined) for the Canon 100-400 and Sigma 150-600.

The error bars show the IQR and the blue marker is the median of the combined median sharpness (wide open) data for each camera.

Things to bear in mind:
1. This is ALL camera types combined, so you shouldn't take the exact values and apply them to your camera results, but it does give a rough idea of the range and sharpness across the range

2. I hacked this together quickly, so I make no promises as to the accuracy of the data! (although I do think it's correct)

3. This is generated from slightly newer data than is current available within FoCal - this new data will be uploaded shortly.

 

[bluenoser1993]

OK, once again I'm not completely sold on the comparability of the results. I was focusing more on my 100-400II with poor and better technique, but in getting ready to use my 135L this weekend I looked back at the test results. I had only paid attention to the AFMA setting at near and far distances, which only changed by 2. However, at 3.3m distance the QoF was 1520 and at 10.7m distance the QoF was 1820. Tested within minutes with the same lighting, target, setup, etc on my 5Ds. I don't have access to FoCal's data base, so don't know how this compares to the average range, but it does demonstrate how much spread is possible with the same camera, lens, target, lighting at different distances. It would appear that the algorithm has overcompensated for the softer image result because of the longer range, at least in this case.

Rich, I was doing this intentionally to see the AFMA required for longer range work and using my camera in crop mode. Because of the calculated distance, would this be a result that wouldn't make it into the data FoCal shares?

EDIT to change 10.3m to 10.7m, shouldn't have trusted my memory.

 

[bluenoser1993]

OK, once again I'm not completely sold on the comparability of the results. I was focusing more on my 100-400II with poor and better technique, but in getting ready to use my 135L this weekend I looked back at the test results. I had only paid attention to the AFMA setting at near and far distances, which only changed by 2. However, at 3.3m distance the QoF was 1520 and at 10.7m distance the QoF was 1820. Tested within minutes with the same lighting, target, setup, etc on my 5Ds. I don't have access to FoCal's data base, so don't know how this compares to the average range, but it does demonstrate how much spread is possible with the same camera, lens, target, lighting at different distances. It would appear that the algorithm has overcompensated for the softer image result because of the longer range, at least in this case.

Rich, I was doing this intentionally to see the AFMA required for longer range work and using my camera in crop mode. Because of the calculated distance, would this be a result that wouldn't make it into the data FoCal shares?

EDIT to change 10.3m to 10.7m, shouldn't have trusted my memory.

Just to demonstrate that this isn't a sample of one, the results with the 1.4X attached show the the same. At 4.7m the QoF is 1450 and at 10.7, the QoF is 1600.

So with all the same test conditions, my body and lens score a better QoF with the 1.4X attached at 10.7m than it does as a bare lens at 3.3m.

Again, these distances were not chosen to compare QoF, the 25x = 4.7m for standard use and the 50x plus 1.6 crop factor = 10.7m for long range use to determine if custom AFMA settings were required. I just mention it because the QoF results seem pertinent to this discussion.

EDIT: I PM'ed Rich my serial number and test date/time in case it would allow him to look up my tests and add any insight to the results and hopefully post them here.

EDIT: changed the near distance with 1.4x attached to 4.7. I shouldn't post so late at night!!

 

[AlanF]

FoCal has an on-line calculator for minimum distance. For the 5DS, the value for a 135mm lens is 4.35m (4.2m for a 120mm, 4.5m for a 150mm) and with a 1.4xTC, just under 6m. So, at 3.3m you are well under their recommended minimum values, which might mean you are out of range.

A couple of other points

How many repeat runs do you make at each distance? All experimental measurements have a spread of mean values and a standard deviation. As an experimental scientist whose work depends on accurate measurements, I am anal compulsive about repeat runs, and even when measuring my blood pressure at home I do at least 5 repeats and calculate the mean and standard deviation. I find the QoF values do vary on repeat measurements at the same distance. You'll see on the chart in the thread for my lenses, I reported a range of values for each on each body.

The differences with distance could be real. Some lenses do change their MTF values at different distances. It gets reported, for example, that the Tamron 150-600mm is sharper at long distances while the converse is true for the Nikon 200-500mm (from memory). Lensrentals do their MTFs on "Olaf" at infinity whereas Imatest used by most testers is closer up, which some have suggested accounts for different review results.

 

[Alex_M]

All Sigma Art lenses perform poor at MFD (approx. 15% less than peak sharpness) and absolutely worst at infinity (approx. 20% less than peak sharpness) with peak sharpness achieved at distance to Focal target being approx. x30 the focal length of the lens.

FoCal has an on-line calculator for minimum distance. For the 5DS, the value for a 135mm lens is 4.35m (4.2m for a 120mm, 4.5m for a 150mm) and with a 1.4xTC, just under 6m. So, at 3.3m you are well under their recommended minimum values, which might mean you are out of range.

A couple of other points

How many repeat runs do you make at each distance? All experimental measurements have a spread of mean values and a standard deviation. As an experimental scientist whose work depends on accurate measurements, I am anal compulsive about repeat runs, and even when measuring my blood pressure at home I do at least 5 repeats and calculate the mean and standard deviation. I find the QoF values do vary on repeat measurements at the same distance. You'll see on the chart in the thread for my lenses, I reported a range of values for each on each body.

The differences with distance could be real. Some lenses do change their MTF values at different distances. It gets reported, for example, that the Tamron 150-600mm is sharper at long distances while the converse is true for the Nikon 200-500mm (from memory). Lensrentals do their MTFs on "Olaf" at infinity whereas Imatest used by most testers is closer up, which some have suggested accounts for different review results.

 

[AlanF]

All Sigma Art lenses perform poor at MFD (approx. 15% less than peak sharpness) and absolutely worst at infinity (approx. 20% less than peak sharpness) with peak sharpness achieved at distance to Focal target being approx. x30 the focal length of the lens.

FoCal has an on-line calculator for minimum distance. For the 5DS, the value for a 135mm lens is 4.35m (4.2m for a 120mm, 4.5m for a 150mm) and with a 1.4xTC, just under 6m. So, at 3.3m you are well under their recommended minimum values, which might mean you are out of range.

A couple of other points

How many repeat runs do you make at each distance? All experimental measurements have a spread of mean values and a standard deviation. As an experimental scientist whose work depends on accurate measurements, I am anal compulsive about repeat runs, and even when measuring my blood pressure at home I do at least 5 repeats and calculate the mean and standard deviation. I find the QoF values do vary on repeat measurements at the same distance. You'll see on the chart in the thread for my lenses, I reported a range of values for each on each body.

The differences with distance could be real. Some lenses do change their MTF values at different distances. It gets reported, for example, that the Tamron 150-600mm is sharper at long distances while the converse is true for the Nikon 200-500mm (from memory). Lensrentals do their MTFs on "Olaf" at infinity whereas Imatest used by most testers is closer up, which some have suggested accounts for different review results.

The thought has crossed my mind that a lens manufacturer might be tempted to optimise their lenses for distances used by most reviewers for their test charts. But, of course they wouldn't.

 

[bluenoser1993]

FoCal has an on-line calculator for minimum distance. For the 5DS, the value for a 135mm lens is 4.35m (4.2m for a 120mm, 4.5m for a 150mm) and with a 1.4xTC, just under 6m. So, at 3.3m you are well under their recommended minimum values, which might mean you are out of range.

A couple of other points

How many repeat runs do you make at each distance? All experimental measurements have a spread of mean values and a standard deviation. As an experimental scientist whose work depends on accurate measurements, I am anal compulsive about repeat runs, and even when measuring my blood pressure at home I do at least 5 repeats and calculate the mean and standard deviation. I find the QoF values do vary on repeat measurements at the same distance. You'll see on the chart in the thread for my lenses, I reported a range of values for each on each body.

The differences with distance could be real. Some lenses do change their MTF values at different distances. It gets reported, for example, that the Tamron 150-600mm is sharper at long distances while the converse is true for the Nikon 200-500mm (from memory). Lensrentals do their MTFs on "Olaf" at infinity whereas Imatest used by most testers is closer up, which some have suggested accounts for different review results.

I just edited my post regarding the 1.4x attached to reflect 4.7m at the near distance, I mistakenly used the same 3.3m number that was used for the bare lens test. I just looked at the recommendations on their site and was surprised by the minimum they recommend. I have read in many places that the recommended distance for testing lens is 25x to 50x the focal length. Even on FoCal's site , if you click the more info that is below their distance calculator they actually say you can go as low as 20x for longer focal length (defined by them as 300mm or more). My near distances were 25x.

 

[FoCal Rich]

Hi

I have seen this and the PM and will reply soon (very busy with release testing at the moment)

Rich

 

[bluenoser1993]

OK, once again I'm not completely sold on the comparability of the results. I was focusing more on my 100-400II with poor and better technique, but in getting ready to use my 135L this weekend I looked back at the test results. I had only paid attention to the AFMA setting at near and far distances, which only changed by 2. However, at 3.3m distance the QoF was 1520 and at 10.7m distance the QoF was 1820. Tested within minutes with the same lighting, target, setup, etc on my 5Ds. I don't have access to FoCal's data base, so don't know how this compares to the average range, but it does demonstrate how much spread is possible with the same camera, lens, target, lighting at different distances. It would appear that the algorithm has overcompensated for the softer image result because of the longer range, at least in this case.

Rich, I was doing this intentionally to see the AFMA required for longer range work and using my camera in crop mode. Because of the calculated distance, would this be a result that wouldn't make it into the data FoCal shares?

EDIT to change 10.3m to 10.7m, shouldn't have trusted my memory.

Just to demonstrate that this isn't a sample of one, the results with the 1.4X attached show the the same. At 4.7m the QoF is 1450 and at 10.7, the QoF is 1600.

So with all the same test conditions, my body and lens score a better QoF with the 1.4X attached at 10.7m than it does as a bare lens at 3.3m.

Again, these distances were not chosen to compare QoF, the 25x = 4.7m for standard use and the 50x plus 1.6 crop factor = 10.7m for long range use to determine if custom AFMA settings were required. I just mention it because the QoF results seem pertinent to this discussion.

EDIT: I PM'ed Rich my serial number and test date/time in case it would allow him to look up my tests and add any insight to the results and hopefully post them here.

EDIT: changed the near distance with 1.4x attached to 4.7. I shouldn't post so late at night!!

Quoted my post so all numbers would be in one spot. As suggested, I ran another set of tests on the 135 with and without the 1.4x, and I added distance to the near range to meet FoCal's minimum recommendation (though this puts it out of the intended range for portrait work). Bare lens there was improvement at both ranges, but the gap narrowed a bit:
At 10.4m = 1950
At 4.4m = 1700

With 1.4x attached:
At 10.4m = 1630
At 6m = 1520

Bare lens, total spread of 4 tests is 1520 - 1950
with 1.4x, total spread of 4 tests is 1450 - 1630

(note, there were in fact higher spikes of QoF values but I took the representative value from the FoCal report, not the highest spike)

A second finding: this second round of testing, similar to the 100-400II last week, was done with greater care. While the amount of care I took with the 100-400 had very little affect on the QoF, it had considerable affect on the outcome with the 135L. While some care has been taken by FoCal to equalize QoF value results by measuring test conditions and factoring for them with the algorithm, it doesn't work equally well in all cases. I'm afraid I'm back to believing the data isn't as reliable as we'd like it to be for comparing our lenses against. In saying that, I don't have the data of the 135 to compare to my results, perhaps Focal has found that lens to be less predictable in focus accuracy, though that would contradict many reviews.

 

[takesome1]

I'm afraid I'm back to believing the data isn't as reliable as we'd like it to be for comparing our lenses against.

What exactly do you think this comparison is going to tell you about your lens?

 

[bluenoser1993]

I'm afraid I'm back to believing the data isn't as reliable as we'd like it to be for comparing our lenses against.

What exactly do you think this comparison is going to tell you about your lens?

It's being marketed as a means to compare your lens against all the other users lenses. There is a claim that the results are normalized to make this comparison possible. The trials I ran with one lens seemed to back that up, but the results from another lens do not support the claim.

 

[takesome1]

I'm afraid I'm back to believing the data isn't as reliable as we'd like it to be for comparing our lenses against.

What exactly do you think this comparison is going to tell you about your lens?

It's being marketed as a means to compare your lens against all the other users lenses. There is a claim that the results are normalized to make this comparison possible. The trials I ran with one lens seemed to back that up, but the results from another lens do not support the claim.

Unless I am missing something here, and Rich can correct me, the data is collected from the Quality of Focus report that you can create on your computer. This tests relies on the AF system of your camera. Your AF system may be precise and accurate, or it may not, but your camera body and its AF system can have an impact on this test. So how you test and what body you test with can all sway the results. This data gives you a body and lens comparison, not just lens.

If you are testing just the lens the Aperture / Sharpness report would be a better judge of the lens. In this test you do not have to use the AF system, you can manually focus in live view and run the test. As AlanF mentioned in a previous post about repeating tests, if you run this test many times you will have a good understanding how your lens performs at various apertures and if the information is available how it compares to the same camera and lenses of others. I find this test the most useful.

I do not think either test is a good indication of a great lens since the lower and upper 25% of data is removed. I think an average lens might be indicated and bad lens yes it would.