Thank you for all the feedback so far. Keep in mind I am shooting with a Canon 5D Mark I. Just from reading the comments I gather the main difference in real world situations is sharpness and bokeh, realistically how big is the difference in sharpness?
Also is there a difference in the IS?
I think these opinions always boil down to whether an image is pixel-peeped or not. I had the Mk I for years before upgrading to the Mk II. When the Mk I was around, many people said how awesome it was and that Canon couldn't improve on it. I never believed that because I bought two copies and even compared the one I chose to a friend's copy. None were sharp at the 100% level. I've got a fair amount of glass and of all the lenses I have used, the 70-200 Mk I was probably the second softest lens (at 100% at 2.8 or wide open) I've used a lot. The 16-35 wide open is pretty crappy too (at 100%). The other lens I'm not crazy about is the 24-105. I do use that lens a lot because it has IS and is great for taking snapshots of the young kids. When I want an image with high IQ, I will use something else. The new 70-200 Mk II, is not as sharp as my copy of the 85L, but it's close enough for me that I would use it for important photos.
Now if you are not the kind of person who views images at 100%, then you likely won't notice a difference in sharpness between the Mk I and the Mk II. If you edit at 100% or you do pixel peep (which I proudly admit to), then you will notice what I would call, a HUGE difference when shooting at 2.8.
Anyone who says there isn't a sharpness difference at 2.8, is not viewing at 100%. I can almost guarantee it.
I would even go so far as suggesting you save for the Mk II if you can't afford it now.
The locking lens hood is a huge improvement for me too. For some reason, my hoods often rotate out of lock for me so much that I have to keep checking it. I even used gaffers tape to tape down the hood of my 24-105 hood. You don't have that problem with the 70-200 Mk II.