I had two or three non L versions and sold them all. I like to hand hold them, and could not get consistently sharp images hand held. For many images, it took so long to focus that using AI Servo, I often clicked the shutter before the lens had finally focused, or at least, that what I suspected.
I'm much happier with the "L" as a walk around lens, it focuses quickly, and the Hybrid IS lets me take handheld images that are reasonably sharp.
As long as you are patient and careful, the 100mm USM is plenty sharp, its just not the best choice for someone who is impatient.
Here are some I took with the 100mmL just walking around.
Here is one with the 100mm USM