You have a strong kit, which makes fine-tuning all the more difficult. One of the things that I consider when refining/downsizing my kit is: what can this lens accomplish photographically that THIS lens can't. The differentiating factor, as it were.
In your case, there is nothing the 4.0 version of the 70-200 can do that the 2.8 70-200 IS II can't do (except BE LIGHTER). I know how hard it is to lose the 70-200 f/4 IS (I had to choose between it and the 70-300L). It took me about four months to finally let it go and keep the 70-300L. Why? Because it could do two things the 70-200 f/4 IS couldn't: shoot at 70mm @ MFD better and shoot at 300mm better w/o a teleconverter. Being smaller to pack was just a bonus.
Similarly, the 50mm f/1.4 can shoot at 1.4 which the 40mm pancake can't. The 135mm can shoot at f/2 which the 70-200 f/4 IS can't.
Conclusion: Lose the 40mm pancake AND the 70-200 f/4 IS and get 135/2.0 AND the 50mm 1.4. Fast glass rules!
You know, I'd just avoided picking up a second-hand Sigma 50 1.4 EX, and now you've got me thinking about a fast 50 again. Sigh The problem with liking fast glass (and I do) is wanting primes in lots focal lengths! Yes, losing the pancake might not be a bad idea - it isn't getting much use these days, and a 50 1.4 would offer something much more interesting. Or I could lose the pancake and not replace it - I'd still have the 35 and 85 primes, and the zooms. (Trying to convince myself about I don't "need" a fast 50!)
What I'm still trying to decide is whether I'd enjoy the f/2 of the 135L, and it's small / easy to carry size, enough to make it worth giving up the conventence of the zoom of the 70-200 4 IS as a travel telephoto when I don't want to carry the 70-200 2.8. I hear what you say about it offering something "more" than anything else in my kit since it's a telephoto which goes to f/2.
I know I'm lucky to have a pretty good kit (I'd say excellent but then I look at the gear lists of a few other CR members!!) and perhaps i already have the best kit for me. Not sure which way I want to jump with this one. Equally, I'm not keen to add to the size of my kit. Hhmmm. I think I need to find a way to try out a 135L for a little while so I can experience it for myself.