I have a lot of love for my 35L and 135L. I enjoy shooting with a small 28 1.8 and with 85 1.8. My 24-105 is a tool. No mad love, no pasion. However, I use it a lot when i work with strobes. This is practically my workhorse for home studio work. This lens for studio plus primes for everything else. So yeah, I'm not planning to sell mine anytime soon. No love, but a very strong practical relationship.
I've got the 24-105L and love it. I also have the 50/1.4 (along with a bunch of other oddball 50-58mm vintage glass) and the new Sigma 35/1.4 Art lens. Here's my advice: If you want a good prime and you have to dump one lens to add another, get the Sigma 35/1.4 and dump your 50/1.4. The 35 is a bit wider, but at f1.4 there is no comparison to the 50. Sharp sharp sharp, corner to corner. I have to stop down to about 5.6 on my 50 before it gets good, and then you're into the same territory as the 24-105. The 35 has great sharpness and good bokeh, much better than the 50/1.4.