Yes, you will most definitely see a difference. I don't own the non-L variants, but I have tried to use both 50mm L and 85mm L for street photography and I tend to shoot at late hours, so I usually only use f1.2 to f1.8 apertures. Using the faster L primes not only provides you with even better bokeh, but it also allows you to freeze motion, which may become blurred using a slower lens. This is of course only pushing a boundary, but it does make a difference.
There is a significant advantage using a 1D-X or 1Ds Mark III, because of the more powerful battery driving AF motors faster, but that's the only thing to improve compared to using a 5D Mark III.
Many on this forum complain about high cost of particular lenses. I would rather not touch this subject here.