I just clean the front surface of the filter that is on my lens pretty much all the time.I agree...I use only the best filters, though.I am a fan of Lenspen and similar products even for my expensive multi coated filters.
1. Use blower for loose dirt
2. Use brush
3. Use cleaning end of the Lenspen with the Carbon-Laden swab
4. Use brush an/or blower if necessary
Works like a charm and no abrasion.
If someone tells you to use "Lens Tissue"...RUN LIKE HELL!!!!