Why don't you buy both used, see which one you like better and then sell the lens you don't prefer. I have a 70-200mm f/2.8L IS mkii and I'm quite fond of it.
Why don't you check lightroom to see which aperture you shoot at when you take the 70-200mm f/4L out on the town. If it is 200mm 90% of the time... save the cash and buy the prime. If you are constantly adjusting the focal length... then you have your answer.
Thanks for the replies. It is good to know that the IQ is pretty similar for both lenses so now my decision making becomes a bit easier.
There is a deal going down locally where a used 200 prime in excellent condition is available for about $ 500 - $ 600. I think I'll get that and it'll be useful when I need to carry less weight on my hiking / biking trips.
I'll see if I need the zoom once I use this lens. I don't think I could lose too much money if I were to sell it again