I think its more to do with WHAT you're taking, rather than just the act of taking the shots...
Theres a group, I'm a photographer, Not a Terrorist, and its worth looking them up. They've got a card you can print out that lists out your rights, and what to do/not do...