Zooms just dont deliver the same IQ at all focal ranges they offer. Its the culprit of zooms. Theyre ALWAYS a compromise between affordability and IQ. If that doesnt cut it for you then better get primes.
Usually, but not always true: here is a comparison of the amazing 70-200mm f/4 IS showing it holds its own against the best Canon primes, and it isn't that expensive.
And, the 70-200mm f/2.8 IS II is at least as good as the f/4.