Congratulations (sort of) for finding one of the problems with focus stacking on your very first try. Like many other things, there is no fix for all situations. But once you know what you are dealing with the you have some options for mitigation. This was written with the idea of having inline images. That option does not seem to be supported
So what happened?
Your samples are labelled 1 to 3 and they are closest to and furthest away respectively from the lens. When the edges in layer 1 are sharp, the lower petal in layer 3 is out of focus. When layer 3 is in focus, layer 1 is blurry. But in addition to a general blur, out of focus portions of layer 1 have also “bloomed” to a larger size than in layer 1. So now we have a problem. Whenever layer 3 is sharp, some portion will always be blocked by the bloom in layer 1. Stacking algorithms have trouble making up data along the edge in a convincing manner.
Do others have this problem?
My lab has a dedicated Leica imaging system that has features to minimize the problem but still includes some tools for a manual work around.
The first image exhibits some of the same issues in your sample. It was created from a stack of 23 images each with a 0.1180mm Dof. The Leica system uses telecentric lenses to reduce magnification changes as a function of distance. The single most important issue for better results is to leave the focus constant (tape it off) and move the camera on a macro rail.
The next two images are at the beginning and end of the stack. The white dashed lines indicate the in-focus region. Notice the blooming circled in orange.
The Leica system also calculates and displays a depth map and confidence map so you can see what is guiding the algorithm.
If necessary, you can fix problems manually by drawing a mask in filling in problem areas from regions of good focus.
By properly using all the tools, excellent results can be obtained for really small objects.
What does this mean for Photoshop?
Your post did not ask “How can I get my lab system to make better images?”, but I thought it might be interesting to see that even $50,000 systems sometimes struggle with the same issue you discovered.
Now that you know the cause, it becomes easier to make a plan that works for you. When the subject size is constant or growing as a function of object distance, then the automated stacking algorithms work pretty well. When the subject geometry is getting smaller as a function of distance, then blooming can become an issue. To minimize these effects:
1) Look for an angle that minimizes Z-height variation.
2) Optimize image count to the smallest number by critically selecting focus planes.
3) Use depth of field to reduce image count requirements.
4)
Fix the focus and instead move the camera for each image.
5) Stack in Photoshop with blurry images on top and reveal sharper layers below with erase tool.
I’ve done manual Photoshop editing to create stacked images and generally the technique involves erasing part of an image to reveal a sharper image from the layer below. When the two images above are stacked, the resulting image looks good.
Focus stacking is a great tool, but the source images need to be controlled for good results.
Hope that helps.