If you believe that filters protect your lens from impacts, you need to look on Youtube at the videos of people intentionally abusing lenses. What you'll find is that it takes a LOT of impact to even scratch the outer element of a lens (there is at least one where a guy is hitting it with a hammer with no apparent damage). Try doing that to the extremely thin piece of glass in your filter and you can expect it to shatter.
If you're talking about protecting it from dust and dirt, I can see that to a degree, but as others have pointed you should be able to plan and prepare for those scenarios and use them as necessary.
Under ideal shooting conditions, even a cheap filter may not affect IQ much. It's when you get into less than ideal situations, i.e where there is a potential for lens flare, where filters can really start to affect IQ. And buying a B+W filter doesn't automatically mean it won't affect IQ. If you want to use one want to minimize the effect on IQ, you should really use a multi-coated filter (look for "MRC" on B+W filters) but you'll also find that they get expensive quickly.
At the end of the day, you should really always be using a lens hood, both to protect from lens flare and to protect your lens, and if you're doing that in most scenarios a filter isn't going to help much more. However, if you're NOT using your lens hood and ARE using a filter (especially a cheap one), there's a decent chance that at times IQ will be negatively impacted.