FoCal Database for Lens Quality of Focus

[neuroanatomist]

If you want to use this number to compare the quality of a lens you need to have all things equal. Here are a few factors I know of that you didn't mention. Matt paper vs glossy for a target. Printing with a printer capable of printing photo's vs a normal laser jet. Target position and placement being square to the set up. Positioning of lighting, is the lighting direct or indirect at target. Type of lighting, I have several halogens that cast shadows.

^^This.

That's why, IMO, Reikan initially and correctly took the position they stated: "The absolute QoF value is unimportant, so you cannot compare the numbers from one test to another." It was only when they could make money from comparing absolute QoF numbers that their position changed. They're certainly not alone in changing their tune when money is at stake.

Nikon, until 2013: "Fluorite cracks easily and messes up focusing, so we developed ED glass because it's much better."
Nikon, post-2013: "Flourite is great becuase it optimally corrects CA and makes a lens lighter."

 

[AlanF]

As a scientist, I always have more unpublished data to present to the referees to counter their arguments.

It also seems that as a scientist, you initially 'published' your best examples, not representative examples.

I know you you have a tongue in cheek emoticon. But, for the record, I "published" initially none of my values, just the values from the FoCal database. I then "published" my own data that overlapped with the FoCal database in the context of your arguments that poor technique would skew the numbers down - my less than perfect technique gave above average results. Bluenoser has elegantly shown that on going from about the worst conditions of having a crude target pasted on to a vibrating refrigerator door, illuminated by something a bit brighter than a candle, with his camera mounted on what Mt Spokane describes as a floor ringing like a bell to a using sophisticated techniques that there is hardly any change in measured QoF.

 

[bluenoser1993]

Well, I hope someone can explain these results. I did screenshots of the reports at the peek AFMA value so you can see the images side-by-side that are giving the QoF. You can see the extra light in the report, and shutter speed. What you can't see is the heavy weight hanging on the tripod, tripod on basement floor (cement), central heat shut off (no vibration), tripod legs forced into wider spread at floor to reduce wobble, target on solid wall, target larger and of much better print quality, FoCal delay after mirror lock increased to 3 seconds, more time waiting after each AFMA change on body before continuing.

400mm and 140mm were done at almost same distance of 10.1 - 10.3 meters, I didn't have room to get any further away so the new 560mm was done at 10.1m where as the original was done at 13m.

It made sense to see the 400mm QoF increase (though I had hoped for more), but then I was puzzled to see a reduction of QoF at 560mm, and then essentially no change at 140mm.

The one thing I wonder, does FoCal use the declared target size (and hence calculated distance) as a part of the calculation of the QoF? I assumed having the target size increase from 116mm to 209mm would have improved the QoF because of the improved image captured, but maybe it's accounted for?

EDIT - left image is the new (better technique) attempt in each focal length below.

I can say they are absolutely meaningless when in comparison to each other.
Different lighting produces different results. The size of the target and the distance modify everything. The examples you show are from either different distances, the same distances or different targets.

I think this is a good example of why the averages that focal provides can not be relied on as a guide of lens quality.

If you want to use this number to compare the quality of a lens you need to have all things equal. Here are a few factors I know of that you didn't mention. Matt paper vs glossy for a target. Printing with a printer capable of printing photo's vs a normal laser jet. Target position and placement being square to the set up. Positioning of lighting, is the lighting direct or indirect at target. Type of lighting, I have several halogens that cast shadows.

I realized the results wouldn't be meaningful to compare, particularly if trying to compare different equipment. The point was that it is the same equipment, the first run was with very poor technique and the second run was done by changing the aspects I could in an attempt to get the best QoF value with the things I had at hand. I assumed going into the second test that it may shed a little light on the spread of the average, but instead I actually got a reduced value in one case and equal in another. Alan's reply from Reikan seems to confirm the findings of my test, the QoF value calculation is designed to attempt an equal playing field for comparison. I did doubt the comparability, but now think it is more comparable than I thought. A little bummed that my 100-400 scores low.

 

[AlanF]

If you want to use this number to compare the quality of a lens you need to have all things equal. Here are a few factors I know of that you didn't mention. Matt paper vs glossy for a target. Printing with a printer capable of printing photo's vs a normal laser jet. Target position and placement being square to the set up. Positioning of lighting, is the lighting direct or indirect at target. Type of lighting, I have several halogens that cast shadows.

^^This.

That's why, IMO, Reikan initially and correctly took the position they stated: "The absolute QoF value is unimportant, so you cannot compare the numbers from one test to another." It was only when they could make money from comparing absolute QoF numbers that their position changed. They're certainly not alone in changing their tune when money is at stake.

Nikon, until 2013: "Fluorite cracks easily and messes up focusing, so we developed ED glass because it's much better."
Nikon, post-2013: "Flourite is great becuase it optimally corrects CA and makes a lens lighter."

Flourite is more likely to be used in baking cakes.

Takesome1 comments are all hypothetical. You, as a fellow scientist, know full well that he should do experiments to test the actual magnitudes of his hypotheses before pronouncing them like Newton's Laws. I have done experiments under various conditions to know what he states are just second order effects, as now found under more extreme conditions by Bluenoser.

 

[takesome1]

If you want to use this number to compare the quality of a lens you need to have all things equal. Here are a few factors I know of that you didn't mention. Matt paper vs glossy for a target. Printing with a printer capable of printing photo's vs a normal laser jet. Target position and placement being square to the set up. Positioning of lighting, is the lighting direct or indirect at target. Type of lighting, I have several halogens that cast shadows.

^^This.

That's why, IMO, Reikan initially and correctly took the position they stated: "The absolute QoF value is unimportant, so you cannot compare the numbers from one test to another." It was only when they could make money from comparing absolute QoF numbers that their position changed. They're certainly not alone in changing their tune when money is at stake.

Nikon, until 2013: "Fluorite cracks easily and messes up focusing, so we developed ED glass because it's much better."
Nikon, post-2013: "Flourite is great becuase it optimally corrects CA and makes a lens lighter."

Flourite is more likely to be used in baking cakes.

Takesome1 comments are all hypothetical. You, as a fellow scientist, know full well that he should do experiments to test the actual magnitudes of his hypotheses before pronouncing them like Newton's Laws. I have done experiments under various conditions to know what he states are just second order effects, as now found under more extreme conditions by Bluenoser.

That's funny. Perhaps it is because I bought one of the early versions of FoCal and maybe you have a new version that is far more refined. Or perhaps it is because of the variations I have seen from one test to the other. Or maybe the hundreds of posters who do not even know why they have soft pictures and the forum refers them to FoCal. It is just not a test that would find meaningful results.

I pulled this from your post, so I am relying on you for the accuracy:

Reply from Reikan:
Yep, the current QoF calculation is designed to provide comparison as much as possible. It's hard to guarantee as test environments can be very different. The more similar the test set up the more likely the results will be directly comparable

I have in the past used FoCal to compare identical lenses on the same body. I have always done this with the same identical set up, at the same distance and if possible on the same day. The last one that I did was several 500mm II's and my old 500mm. Interestingly in this test the version 1 matched the new versions, who were both within a few points of each other. Also while initially setting up the test my results varied by as much as 300.

 

[neuroanatomist]

Perhaps it is because I bought one of the early versions of FoCal and maybe you have a new version that is far more refined.

...

Reply from Reikan:
Yep, the current QoF calculation is designed to provide comparison as much as possible.

Interesting point, and one which caught my eye in Alan's earlier reply. That suggests they've refined the QoF algorithm.

The problem I have with that is they stated 'absolute QoF is irrelevant and not useful for comparisons' at the time of v1.9, then with v2.0 they launched their comparative database...which therefore must have been compiled with data generated from v.1.9 and earlier, when QoF was not to be used for comparisons.

Honestly, I don't have the data to know whether their comparisons useful or not. But based on their own statements, I really question their motivation...

 

[takesome1]

Honestly, I don't have the data to know whether their comparisons useful or not. But based on their own statements, I really question their motivation...

I think it would be useful if they published the top 5%, rather than the average from the masses.
But then if the motivation is monetary the average from the masses will make more people feel better. In contrast if an individual spends $12k on a new 600mm how will he feel when he is below the 5% group.

 

[AlanF]

Honestly, I don't have the data to know whether their comparisons useful or not. But based on their own statements, I really question their motivation...

I think it would be useful if they published the top 5%, rather than the average from the masses.
But then if the motivation is monetary the average from the masses will make more people feel better. In contrast if an individual spends $12k on a new 600mm how will he feel when he is below the 5% group.

The information we need and is most useful is whether the distribution is a standard Gaussian (normal), skewed or binomial, and if a normal distribution the mean and standard deviation. If Neuro's fears are correct, the distribution will be skewed.

Regarding the clientele, it's us geeks that buy and want to know about their lenses. In certain matters, I would be very happy to be in the top 5%. But, for lenses, average is good enough, although better than average is nice. It would be good for Canon to have their own dumbed-down database so everyone thinks they have bought a cracker of a lens.

 

[FoCal Rich]

Hi!

I'm Rich - the lead developer of FoCal, thought I'd chip in with a couple of thoughts

There's quite a lot of sets of data in the comparison information fed back to FoCal, but the main graphed values are typically the IQR (so 25th to 75th %ile range) of filtered data. The filtering is designed to remove results which are obviously erroneous or suspicious. Single user data is aggregated so there's no bias from a single user (even if they run 100's of tests), and there's a threshold for the number of unique cameras that are required to create any particular data set. Note that the comparision shown on the graphs is matched specifically to the camera/lens combination under test.

With regards to the QoF value - up to FoCal 1.9.5 we used an analysis method (called Q7) which was quite susceptible to lighting variations and image content. From 1.9.5 onwards, the analysis method (called Q10d) was changed to be much more robust. There's quite a detailed blog post here: http://www.reikan.co.uk/focalweb/index.php/2014/02/reikan-focal-rgb-analysis/

With Q10d, the QoF value is generally comparable for the same camera/lens combintation. It's even roughly comparable across different cameras and lenses, but there are some limitations. We only use Q10d results to build the comparison data so even though data was uploaded before FoCal 1.9.5, that data isn't considered robust enough to form part of the comparison data.

The building of the comparison data (the filtering and aggregation of raw uploaded data) is being continuously improved, and it's an area which will be receiving some quite considerable attention very shortly.

Hopefully useful information(!)

Rich

 

[takesome1]

Honestly, I don't have the data to know whether their comparisons useful or not. But based on their own statements, I really question their motivation...

I think it would be useful if they published the top 5%, rather than the average from the masses.
But then if the motivation is monetary the average from the masses will make more people feel better. In contrast if an individual spends $12k on a new 600mm how will he feel when he is below the 5% group.

The information we need and is most useful is whether the distribution is a standard Gaussian (normal), skewed or binomial, and if a normal distribution the mean and standard deviation. If Neuro's fears are correct, the distribution will be skewed.

Regarding the clientele, it's us geeks that buy and want to know about their lenses. In certain matters, I would be very happy to be in the top 5%. But, for lenses, average is good enough, although better than average is nice. It would be good for Canon to have their own dumbed-down database so everyone thinks they have bought a cracker of a lens.

Canon does have their own dumbed-down database. In the US if your less than 1 year old lens to Canon and tell them it is soft, they will send it back to you and say it tested within acceptable parameters. It is so dumbed down they just tell you what it says.

The whole conversation gives me an idea. In a forum such as this you could have a competition, or just a comparison, to see who with the same body and lens can produce the highest quality measure. It would be a competition for the Geeks.

 

[takesome1]

Hi!

I'm Rich - the lead developer of FoCal, thought I'd chip in with a couple of thoughts

There's quite a lot of sets of data in the comparison information fed back to FoCal, but the main graphed values are typically the IQR (so 25th to 75th %ile range) of filtered data. The filtering is designed to remove results which are obviously erroneous or suspicious. Single user data is aggregated so there's no bias from a single user (even if they run 100's of tests), and there's a threshold for the number of unique cameras that are required to create any particular data set. Note that the comparision shown on the graphs is matched specifically to the camera/lens combination under test.

With regards to the QoF value - up to FoCal 1.9.5 we used an analysis method (called Q7) which was quite susceptible to lighting variations and image content. From 1.9.5 onwards, the analysis method (called Q10d) was changed to be much more robust. There's quite a detailed blog post here: http://www.reikan.co.uk/focalweb/index.php/2014/02/reikan-focal-rgb-analysis/

With Q10d, the QoF value is generally comparable for the same camera/lens combintation. It's even roughly comparable across different cameras and lenses, but there are some limitations. We only use Q10d results to build the comparison data so even though data was uploaded before FoCal 1.9.5, that data isn't considered robust enough to form part of the comparison data.

The building of the comparison data (the filtering and aggregation of raw uploaded data) is being continuously improved, and it's an area which will be receiving some quite considerable attention very shortly.

Hopefully useful information(!)

Rich

Rich, thanks for the clarification.

 

[AlanF]

Thanks Rich. You have answered the key questions, and our findings fit in with what you say about a new robust statistical analysis, and I think you will have satisfied the critics. I am also pleased that my large number of calibrations aren't distorting the database.

 

[takesome1]

Rich,

one question.

You say you only use the middle range. One would think the reports with the very high QoF values are more likely to be accurate than the low, since it would take both good technique and good lens to achieve the higher QoF.

Is there any chance you will be releasing a report that would show the upper 25%? Or perhaps one that would show the whole spectrum?

I do understand that your primary goal is showing data that indicates if you are in an acceptable range to do an AFMA on your camera, and part of our discussion is using your software to determine how well our equipment is performing.

 

[neuroanatomist]

Thanks for the information, Rich.

...I think you will have satisfied the critics.

Not quite. For example, they stated, "Back in FoCal 1.5, we started collecting data about the results of your tests (nothing to personally identify you, just a few numbers showing how cameras and lenses behave)." But according to Rich's statement above, results up to v1.9.5 aren't terribly reliable. I wonder what fraction of the current database comprises pre-v1.9.5 data (perhaps zero), and I also wonder what fraction of the database comprised pre-v1.9.5 at the time FoCal 2.0 was released.

 

[FoCal Rich]

Is there any chance you will be releasing a report that would show the upper 25%? Or perhaps one that would show the whole spectrum?

This is something we've got on the list. It requires a change to the structure of the data so it won't work with the current version of FoCal., but we will add more detail (probably every 10th percentile as a compromise between detail and data size).

 

[FoCal Rich]

Thanks for the information, Rich.

...I think you will have satisfied the critics.

Not quite. For example, they stated, "Back in FoCal 1.5, we started collecting data about the results of your tests (nothing to personally identify you, just a few numbers showing how cameras and lenses behave)." But according to Rich's statement above, results up to v1.9.5 aren't terribly reliable. I wonder what fraction of the current database comprises pre-v1.9.5 data (perhaps zero), and I also wonder what fraction of the database comprised pre-v1.9.5 at the time FoCal 2.0 was released.

It's slightly lost in the detail, but in my original post I mentioned that we only build comparison data using the Q10d results - which is only data since 1.9.5 with the new analysis method.

We have done some quite detailed profiling of the performance difference between the Q7 and Q10d analysis and there is some useful data to be obtained from the early results, but as it relates only to older cameras (2013 and before) it's less important than focusing on the newer data.

 

[neuroanatomist]

Sorry, Rich...missed that. Thanks!

Also appreciate the conservative approach of using the interquartile range.

 

[AlanF]

I think we have had a very good robust discussion and have come to a consensus, which is what we should be doing. The database may not be perfect but the uncertainties do not stop it from being useful.

All my dealings with Reikan convince me that it is an ethical company and very responsive. I don't know for how long there will be a market in selling AFMA software if Canon and Nikon introduce auto-AFMA methods and mirrorless erodes the market further. But, software for checking lenses could have a good future for all brands of cameras.

 

[bluenoser1993]

Thanks Rich, it's always nice to have a subject matter expert involved in the conversation. A lot of threads on this forum would be shorter if that happened more often . I think you've cleared up any doubt I had.

In your testing, have you found RAW vs JPEG to have much affect on the QoF?

 

[AlanF]

I have found it makes a difference in my testing for just one case only, 400mm DO II + 2xTC + 5DIV, where RAW has much higher QoF than for jpeg. It makes negligible differences for all other combinations I have used whether RAW or jpeg is used, but jpeg is much faster.