My advice would be that you don't really need the L lenses. If you want to dabble in primes you could try out a 50mm 1.4 for less than $400, and if you find that you really need the extra IQ/build quality of the 1.2L, you could upgrade. Same goes for the 35mm - try out the f/2 version and if you like the focal length, upgrade to the 1.4L if you feel you must.
After owing the 50, both the 1.8 mark1 and the 1.4, I'd never want them after using the 50L. The 35L is an absolute winner too. After you've used the L versions, the non-L stuff is hard to love. The OP has some nice L zooms lenses, suggesting non L primes seems strange. I've never used the 135L.
To the OP, for your budget you could also afford the ef 24mm L Mrk2, Another winner.
I don't disagree with you, and I'm sure the L primes are much nicer than the non-L's. But given the OP stated it is not for professional use and he is not sure if he even needs a prime at that focal length, it seems a bit silly to throw $1500 on a lens he might never use. IMO it would be much wiser to buy the cheap one, test it out, see if he likes the focal length or not, and then consider upgrading if he likes it.