Do people need the extra 100mm that much?
One word: Birds... 400mm is "entry level", even with a crop camera like my 7D, for birding. 300 doesn't really cut it for wild birds. I have a Tamron SP 70-300 f/4-5.6 Di VC USD, and it's a nice, compact lens for travel, but when I'm out birding, it stays home. 100mm isn't a lot of difference, but it helps. There are times I can barely get close enough to a small songbird with the 100-400 @ 400. With the 300, I'd never get close enough.
Do people compare the two lenses or just buy the 100-400L because so many other photographers use it?
I'm sure many do make comparisons, and there are some serious points to consider that could make one choose one lens over the other.
When I was looking, the 70-300L was recently out, and priced similarly to the 100-400 (actually, slightly higher), but it "officially" does not support Canon Extenders. Since I already had a 70-300, and I knew I needed more reach, I opted for the 100-400. Plus, even if it did support the extenders, with a 1.4x, it gets me 420mm @ f/8, where my 7D cannot Autofocus!
Yes, the weight of this lens is an issue for some, but if you want to get good shots of birds, you're gonna be carrying the weight one way or another. Three pounds is pretty lightweight for what this lens does. You're looking at 8 pounds and up for other serious birding options though, with the "big whites". A 400 f/5.6 IS would (well, should...) be lighter, but it's not available.
As for the push-pull design, I don't find that an issue, but I DO wish there was a better mechanism for friction locking the lens in place. The double-ring thing is imprecise at best, and I tend to have it locked someplace, somewhat limiting the versatility. Back in the 80s, when "zoom" was a dirty word (well, more of a dirty word than it is today!) many, many zooms were push-pull. If you want really weird, there were even some where the "tele" end was pulled back, and the "wide" end was pushed forward! My point is, that, while push-pull may be unusual in Canon's lineup (only the 100-400 & 28-300 have it currently), it is not that unusual in the history of zoom lenses. As far as I am concerned, if Canon thinks a new design for the 100-400 is best as a push-pull, that would be fine by me, though (hopefully!) with a better zoom friction system.
Really, what I would most like to see in a new 100-400 is a modern IS and AF (the 100-400 is a FIFTEEN-YEAR-OLD design!) with somewhat improved IQ that might make a II with a 1.4 have similar IQ to the original.
(Heck, if they just dropped modern IS (4 stops, please!) & AF into the current lens, I'd pay them $500 more than what it is now!)
Are people buying now because yes they would love the version II but are scared that it will be to expensive?
Some may be, but I'm not sure that would be the best idea just yet. It's not like the current lens will diasppear. When/If a new 100-400 arrives, the current one will likely be cheaper to buy used, well, at least still cheaper than the new one certainly will be! There will be lots of us who will be looking for ways to fund the version II by selling their version I!