I use them on almost all of my lenses and i am glad i do (do not have enough for all my lenses at the same time).
I was once shooting and a stone got kicked up by a passing car and hit my lens head on, cracking my filter but leaving my lens untouched. I don't know what would have happened if i had not had the filter on, but since then, i have one on! If i want to mount a CPL filter on, i just switch them (or stack them if not too wide). Also, some L lenses need them to complete their weather sealing.
I have some HOYA HDs and B+W MRCs. I have tested with and without and see no difference in the image quality from my eyes, which is what i use and is therefore fine for me!
All my L lenses are 77mm thread. When i need an 82, i will buy a filter accordingly. People who say they see a difference in sharpness (flair is a different matter and can be resolved by removing the filter temporally if so desired) are either using poor quality filters (stupid) or pixel peep to an unhealthy extent (just my opinion).
At the least, i have 1 filter with me at all times that i can put on my lens if conditions warrant it.