In the case of the 70-200/4, the IS version is also sharper, and is weather-sealed.
IS can help at the long end, even for shooting (slowly) moving subjects. The guideline is a shutter speed equal to or faster than 1/focal length will give a reasonable keeper rate to overcome camera shake. For this purpose, it's the FF equivalent, so a 200mm lens on APS-C means you need 1/320 s or so. Similarly, for 70mm on FF, 1/60-1/80 s would do. If you're shooting people, you generally need 1/60 s to freeze the random, involuntary movements that people make (even when holding still). So, you can see the 'justification' (from one point of view) for not needing IS on a 24-70mm lens on FF - that's thought of as a portrait/wedding/event lens, and you'd likely be at >1/60 s anyway. OTOH, at 200mm on APS-C, you may struggle to get enough light to shoot at 1/320, especially with f/4 and the relatively higher ISO noise of APS-C. IS would give you two more usable stops - down to 1/80 s - still enough to freeze people not actively moving), which might be the difference between an acceptable ISO 1600 and an unusable ISO 6400 in some cases (and the f/2.8 IS lens would give you another stop).
Can you live without it? Sure. Is the 70-200/4 non-IS a good lens - absolutely! It's a Canon L-series lens... IS is just one more tool in the toolbox, helpful sometimes, not needed other times.