I'm lucky. This has never happened to me. But I'm not a risk taker. I don't take covert pictures of people who look like they are belligerent, destitute, alcohol or drug fuelled. I'm just not into that sort of photography and that takes a lot of risk away. While I take a lot of photos in public places, it is pretty obvious what I'm doing. If someone is clearly a major part of a photo, I'll get tacit approval beforehand. If they appear to have a problem, I just move on - I'm not looking for problems. I'll often have a camera on a tripod and it will be pretty obvious what the focus of the shot is and people realise that the focus isn't on them. I do have a remote trigger that I use on occassion (admittedly - to get myself in the photo). But how can I be taking a photo of someone when I'm standing 10m away from the camera? Do they think I have some sort of telekinetic power? If so, they should think twice before confronting me! I also tend to only hang out in fun, touristy type places. Everyone has cameras. Everyone's taking photos. Nobody thinks twice about being in the background of someone's photo.
Depending on the situation, I'd be very suspcious of someone who confronted me over taking their picture. Even if I felt pressured to delete the photo, I'm sure I'd find a way to sneek a few more shots of them just in case.