The question is how often you use the 135L at f/2. If you don't use it much wide open, then the 70-200 II can replace it. And because you don't use the 100L for its magnification advantage, the 70-200 II can replace that as well (I find the two similar in IQ). The 70-200 II is a bit heavier and will require better tripod setups. Until recently, I was using a cheap tripod that did not handle the weight of the 70-200 II well, so by default I favored primes over it. It got sand that I could not get out using it on a beach, and the head failed shortly after, which gave me reasons to get a good tripod.

The only big disadvantages of the 70-200II is its size and weight. If you can hold onto the primes while having the 70-200 II for a while, you'll quickly find which ones become expendable.
Privatebydesign also mentions the potential harsh bokeh of the 70-200II. In those cases, I find it helpful to shut off IS. I haven't tested it rigorously, it but stands to reason that it could create a more jittery background with high contrast because the axis of rotation for the camera/user and the IS elements are not the same. He is also right that the 100L does allow you to get closer because of its magnification advantage even if you don't get close to 1:1, which is why I end up carrying the 100L with me even though I have the focal length covered by other options.