This is an interesting thread and I am glad you started it.
In Australia [Where I am from] I am lucky. It is not illegal to take and/or use a persons photograph in a public place regardless of what they ask of you. In relation to lawsuits we are lucky and secure in that regard.
When referring to confrontation from individuals it is always a fun experience. I do a lot of documentary style filming with street sports such as Freerunning. This 'requires' me to enter into places and premises that are not open to the public [Trespassing, my favourite sport]. This has lead to a lot of close calls with everyone from psychotic drugged out women attacking us to security guards and the average Joe who for whatever reason does not enjoy a camera.
Most people are good people, often it is relative to circumstance and environment. If you take a picture of a national icon or landscape from a lookout, people are usually just tourists or travellers and are more than happy with your graphing of photons. Others, not so much. These people are usually by themselves or paranoid [E.G. A mother at a park thinking you are a pedophile. My advice is not to carry around a 800mm f/5.6 near a playground].
The honest and best way to deal with them [other than avoiding them] is to simply apologise, attempt to explain yourself and agree with them. I find this very hard to do [I am far to pretentious] but nonetheless it is the way to go.