This past Saturday, I spent about two hours comparing my 28/1.8 versus the 24-105L.
When you shoot the 28/1.8 at f/4, you get sharper corners, better corner contrast, better flare control, less CA by a huge amount, and from what I can tell... pixel peeping.... at least the same sharpness and contrast and "snap" in the center regions, as compared with the 24-105L. Done with a 5d2.
So, it doesn't have to be L glass to be really good. And granted, the 24-105 is not really to die for in image quality either (though mine is very impressive on the long end).
I can also say, that my 300/4L (non-IS) is as sharp, contrasty and snappy as my 50/1.4, assuming the 50 is stopped down to at least 2.8
The L glass is spectacular though... I'm lusting for a 35/1.4, but I'll probably succumb to a 16-35/2.8 instead.