It really depends on your shooting needs (and partly on your stamina). Indoor events, sports, etc., you want f/2.8 (and for sports, the IS is not really needed). Outdoor use, etc., f/4 is fine, and if you're going to be hiking around with a 70-200mm zoom and other lenses, the f/4 IS may be a better choice. You're paying quite a penalty in cost and weight for that extra stop of light, but if you need it, that's that.
Some people have both the f/4 IS and f/2.8 IS versions, for different purposes. Likewise, I have both the 70-200 II and the 100-400mm (which are about the same size/weight), but for a while I also had the 70-300mm DO for times when I wanted to travel light (and I'm considering the new 70-300 L for the same reason).
One other consideration is that the f/2.8 lens provides a brighter viewfinder image and also allows use of the higher-precision f/2.8 center AF point found on recent camera bodies.