But I expect I would use it 99% at 400 so why not the prime
Have you shot with one enough to know if your expectation is valid? It's likely close, at least. For me, about 15% of shos with the lens are at 150mm and shorter. So, why don't I have the prime? IS and physical length. At 400mm on APS-C (I almost always use my 100-400 with my 7D, since if you're focal length-limited, APS-C is a better choice), you need approximately 1/640 s for a decent handheld keeper rate. Over 50% of my shots with the lens are slower than 1/640 s (mostly 1/250-1/500 s), so I'm getting a lot of benefit from the IS. Also, the 100-400mm gets you to 400mm with a lens that's nearly 3" shorter (when retracted) than the 400mm f/5.6L prime - that means the zoom fits in my Lowepro Toploader 75 AW whereas the 400/5.6 prime would not, similarly for many other bags.
Right on target, same things for me here - zoom is more usable and shorter wich is important too. And 100-400L is not so expensive second hand now, I'll problably go for it for Christmas(if I have the money).