I also wanted to buy a telephoto lens and it was a difficult choice. The unexpensive zooms (70-300 no-L, third party lenses etc.) seem to give really poor images. So I hesitated also between those 2 good lenses (70-200 f/4 IS & 70-300 L) because the other ones (70-200 2.8, 100-400, 300, 400..) are really too big and to heavy to carry with me all along the day (I love hiking).
After lots of searches, I think I will chose the prime 200mm F/2.8 L II. I don't know if you have already considered this cheaper alternative. I have an opportunity to have one second-hand for 600$ (half price of the other two). It's easy to carry (only 13 cm long) and discret (black). It has no weather sealing, no IS but is sharp and good from 2.8 so I'll be able to shoot faster (as fast as the 70-200 2.8 which is twice heavier and half longer) and have a better bokeh for portraits. Comments before my purchase are greatly welcome :-)
I owned the 200/2.8 II for a while, and sold it only after getting the 70-200mm f/2.8L IS II. The 200/2.8 prime is a great lens, and a great value. My only 'complaint' about is is that it's a long focal length to use with no IS - even at f/2.8 you need a fair bit of light to get a 1/200 s shutter speed.