Despite my "ho hum" review of the 40mm it's kind of grown on me, it's a very capable lens and if you don't have a zoom in that focal range it's a no-brainer as a buy.
I do have the 24-105 f/4 but I don't think I have a single "Wow" picture taken with it. It wasn't till I slapped the 50mm 1.8 on on the 6D and got some "Wow" pictures that I realized how special primes can be. My problem with my 50mm is focus accuracy. It is so unreliable. Bought FoCal Pro and every time I run it with the 1.8, it tells me my predicted adjustment is off the chart. If I force it to continue, I get an adjustment of +16...but I digress.
I am happy to play with and get to know my new 85mm for now but at some point, I really feel the need to replace the my 50mm.