I traded in my 100mm non-L last year for the L version. I haven't regretted it. I was also surprised at the amount of money I got for it, more than half of what I paid for it 13 years earlier!
The big difference is smoother AF and having IS, I can't really spot any difference in IQ.
I've pretty much got the camera with the 100mm nailed in place, so IS is a non-issue. However, it's good to know that if the 100mm ever breaks and I can't find the non-L version, I can move up if need be.