One thing not mentioned is lens flare. I had a non-IS 70-200/4 for years, but could not shoot into the lght, especially with the sun in the frame, without having objectionable flare.
I don't have this problem with my 70-200mm f/4 ... perhaps your filter caused the flares (or the lack of one)?
While I keep filters on most of lenses most of the time, I remove them when shooting into the light. Here is the best photo of a series shooting into the rising sun. The subsequent photos when the sun rose further were unusable (and deleted). This was 200mm @ 5.6 on a 5DII.
I don't use filters anymore... I had some good quality ones but I was never satisfied...