I agree the loss of 1-2 stops can be costly in shutter speed or ISO performance in some situation. Originally I had started to shoot with circular polarizing filter whenever I shoot outside in the sun. I since then dialed back on their use to only use them in some situation, mostly for landscape or when I dont need to "freeze" the action.
With the polarizing filter on, I missed several shots where I needed to a higher shutter speed...They are a good tool, but I would not recommend them on all the time.