People have the right to take pictures in any public place
There's a distinction to be made between what's within your legal rights and what's polite or ethical. In the U.S., you have the right to go around like Wowbagger the Infinitely Prolonged and insult everyone you meet, so long as you don't do it in a way that amounts to fighting words. It is, however, extremely uncivil to do so. Regardless of your rights, it's uncivil to take photos of people who don't want to be photographed. The world is not your modeling agency, and its people are not your hired "talent." Don't treat people as mere scenery in the theater performance that is your life.
That's true also, but different from what I was trying to get at. When someone comes up to a photographer and demands that you do not take pictures of them, their kids, their dog, etc. and/or demands to see your photos they are acting as though it's their right not to be photographed when in fact the opposite is true. Legally, the photographer is not prohibited from taking pictures in public. Harassing someone though by interfering with them, cursing at them, threatening, assaulting, damaging their gear (e.g. hollywood celebs) is in fact prohibited. The point is people tend to think they have rights they don't actually have and then get irate when they think you're violating those perceived rights.
Social norms are a different matter. People do feel uncomfortable being photographed and if I noticed someone specifically photographing me whether with a long lens or a smartphone I would feel uncomfortable. If they kept it up I might ask them to stop but would more likely just walk away. If they followed me and continued photographing me specifically now they might be crossing a legal line in the sense of harassing me.
So, yes I get your point but I was specifically talking about rights and the law, not social norms.