I have a 5d (mk I), a 24-105 as walk-around lens, a 50 1.4 for indoor use and a 17-40 as UWA which I will use this summer for my roadtrip in Iceland.
I also wanted to buy a telephoto lens and it was a difficult choice. The unexpensive zooms (70-300 no-L, third party lenses etc.) seem to give really poor images. So I hesitated also between those 2 good lenses (70-200 f/4 IS & 70-300 L) because the other ones (70-200 2.8, 100-400, 300, 400..) are really too big and to heavy to carry with me all along the day (I love hiking).
After lots of searches, I think I will chose the prime 200mm F/2.8 L II. I don't know if you have already considered this cheaper alternative. I have an opportunity to have one second-hand for 600$ (half price of the other two). It's easy to carry (only 13 cm long) and discret (black). It has no weather sealing, no IS but is sharp and good from 2.8 so I'll be able to shoot faster (as fast as the 70-200 2.8 which is twice heavier and half longer) and have a better bokeh for portraits. Because it's a prime, it's also pretty good with a teleconverter (I think I will buy the small 1.4x PRO kenko to have a 280mm F/4) and/or won't hesitate to crop a bit if necessary (I'm not a pro and don't print big posters...) to shoot wildlife.
Comments before my purchase are greatly welcome :-)