I guess my thinking would be that while your current lenses can't do what the 85L can, they can at least get most of the way there (you can step back and shoot 150 f/2.8 and get separation, etc). The TS-E does something none of your lenses can do with the T/S, but, it also can operate as a really sharp Zeiss-like prime as well.
Guess it depends what you think you'd use more of. If you tend to shoot people, street, action, etc more, than the T/S might not get used like your 24-105. If you are out landscaping or shooting cities, it might be the only lens you use