I have both lenses.
I was considering selling the 50mm (to fund the 85mm L) but I've now decided to keep both the 40 + 50.
In my opinion the 40mm IQ is better than the 50mm. OK, you're missing out on some f-stops but the 40mm for me just 'works'. I use it on a 5D and a 7D - on both it's fantastic. I'm keeping the 50mm for those times when the light is really bad! (I shoot weddings - some venues are just so dark).
I had the 35mm f2 but that was just a 'toy', I wanted the 35mm L but just couldn't justify the cost, then the 'forty shorty' came along and I totally love it.
A quick few test shots with the 40 shorty here (they are not that exciting though!):http://rhysbaker.com/blog/index.php?showimage=1234