Whenever you use a filter, it is going to affect you image quality. Fact of life. I keep filters on most of my lenses, unless it has a recessed front element like the 50 1.4. I like not having to worry about fussing with lens caps when i need to switch lenses real fast at a wedding. UV filters let me not worry about it.
What do you mean with a recessed front element? Also, I have always used a Kenko Pro 1 W filter on my 50 1.4, and have never had any issues. Are you saying that the IQ will be better if I remove it? (so far I have had nothing but good thing to say about this particular lens).
Good quality filters will have a very small effect on IQ. But one danger of filters is creating reflections from certain light sources. Example:
When I say recessed front element, i mean that the front lens is set back in the lens, rather than flush with the filter thread. Example of recessed: http://i468.photobucket.com/albums/rr48/bullitt411/Siggy%2010-20/Tokina/DED_3739.jpg
Not recessed: http://www.yacart.com/blog/wp-content/uploads/2011/12/canon_ef_70_200mm_f2.8_l_is_usm_lens.jpg
When the lens element is set farther back in the body of the lens, it is more protected and unlikely to be damaged.