Assuming I needed the extra reach of the 70-300 L I bought one, but must have received a bad copy - sometimes it was no sharper (sometimes it was distinctly less sharp) than my 70-300 non-L IS, so I returned it.
Even with Canon lenses, there are bad copies - look at the iso crops for the 70-300L @the-digital picture, they got another copy because the first one was bad: http://www.the-digital-picture.com/Reviews/ISO-12233-Sample-Crops.aspx?Lens=738&Camera=453&Sample=0&FLI=0&API=0&LensComp=738&CameraComp=453&SampleComp=1&FLIComp=0&APIComp=0
The 70-300 non-L actually is also ok iq-wise except for the long end, so maybe that's why you didn't see a difference. But of course the L has better is, faster & silent non-micro usm af and much better build quality.
Concerning the 70-200/4: It is certainly a good lens and a tad sharper than the 70-300L while keeping constant aperture - but whenever I see it I find it strangely long for something you also shoot at 70mm. Of course internal zooming is technically superior to the telescope design of the 70-300L, but handling & transport imho is simply better with a shorter lens.