Weird - I shoot video constantly on streets and public places, and include people all the time. I'm not inconspicuous - I've got a huge video tripod with a fluid head on it. Nobody, other than drunks and crazy people has ever said a word in all the years I've been working. Occasionally people ask about my project, but they are friendly and interested.
I think the tripod has a lot to do with it. It's so big, the photographer is perceived as working openly, and as a pro.
My suspicion is the public has a certain perception of small camera might be spy, big camera might be paparazzi, but the tripod is reserved to studio or news photographers (= good old pros).