If you read the Christian Bible, you'll see that any time an angel appears to a human, the first thing they say is "Fear not."
That's important advice for anyone taking pictures in public. And I especially counsel that you do not heed the words of posters here who are ill-informed, don't seem to understand the concept of a free society and apparently live their lives in fear. The only time you would need a release for someone you've photographed in a public place is if the image will be used for commercial purposes.
Your question is vague so I'm not sure what the answer is for you. To say you're "...just pointing the camera in general directions without focusing on any one person in particular" is a bit suspect in itself. A camera requires focus, and I don't know anyone who takes pictures randomly without some point of focus. If that's all you're doing, get a pinhole camera -- nobody will even know what it is.
I do a lot of photography in public places, and I have general guidelines I follow. You'll find accomplished street photographers will all tell you generally the same thing.
1. Fear not. If you look tentative, it triggers apprehension in other people. The merely assertive may ask what you're doing. The aggressive and/or angry types may threaten you or at least try to intimidate you. If you're out there to take pictures, look like you know what you're doing and go about it in a businesslike manner. Don't look like you're afraid or like you're doing something you're not supposed to be doing.
2. Dress appropriately. If you're dressed in some extreme fashion, it calls attention to yourself for reasons that may make the camera suspicious. Military fatigues and leather are great for photographing streets scenes in the Castro -- it's not going to look so good shooting around Wall Street. Be aware how you're presenting yourself in both dress and manner.
3. Perhaps the most important thing to remember is that your best defense is a smile and a casual, personable demeanor. If you're not comfortable talking with strangers, you'll have to overcome that. You may want to explore the science of body language. I know and use gestures and mannerisms that communicate I'm not a threat. But the smile is the most powerful force you have -- use it freely! And keep smiling no matter what!
4. One street photographer I know uses lies if someone challenges him -- whatever it takes. If you're young, say you're doing an assignment for your teacher in class. Tell them you're scouting the area because you want to propose to your girlfriend here one day soon. Tell them you're looking for a place to shoot a big bikini layout for some magazine. If you've taken a picture of a person tell them you did it because they look so great or you love their clothes -- flattery can't be overplayed. Hell, tell them Walt Disney sent you. I've never had to use this tactic, but I've got it ready if I need it. Oh, and don't use this one with the police. If the police want to know what you're doing, be honest.
5. As someone already said, take someone else with you. It doesn't even have to be a photographer. People who will challenge one person rarely want to take on two people.
6. If someone tells you they simply do not want you to have their picture, smile and delete it for them. Show them as you delete it. Whatever it takes to keep the peace. But keep smiling. And for years to come you can tell the story of how you took the greatest photo ever made by man, and you had to delete it. Like all good stories, over the years it will be embellished, and eventually you'll have done it while the person held a "Dirty Harry" 44 magnum at your head!
7. One thing I do that I've never seen anyone do is carry what I call "Who the hell are you?" cards. They're standard business cards with a Web address, phone number, etc. I hand them to anyone I talk with on the street. (No physical address. Use a disposable cell phone if you don't want to use a real number.) Often I invite people to look at pictures I've taken of them on the Web and contact me if they want to use them or have prints -- making sure to say I'm not selling anything. Very few people have ever contacted me. What I've found is the mere existence of something they think is a business card instantly makes me legitimate. It's silly, as you can get a box of decent business cards for $10 from Vista Print, but it sure works. Again, whatever it takes to keep the peace.
8. Be aware of children. Nothing gets people excited faster than thinking a stranger is photographing children. While it is plainly legal in the U.S., to take pictures of children in public places, we've become a paranoid society, and it's better to self-censor yourself a bit in this regard.
9. Know and understand your legal rights. Here's the ACLU guide for photographers. This is applicable in the U.S.
Taking photographs of things that are plainly visible from public spaces is a constitutional right – and that includes federal buildings, transportation facilities, and police and other government officials carrying out their duties. Unfortunately, there is a widespread, continuing pattern of law enforcement officers ordering people to stop taking photographs from public places, and harassing, detaining and arresting those who fail to comply.
Your rights as a photographer:
1. When in public spaces where you are lawfully present you have the right to photograph anything that is in plain view. That includes pictures of federal buildings, transportation facilities, and police. Such photography is a form of public oversight over the government and is important in a free society.
2. When you are on private property, the property owner may set rules about the taking of photographs. If you disobey the property owner's rules, they can order you off their property (and have you arrested for trespassing if you do not comply).
3. Police officers may not generally confiscate or demand to view your photographs or video without a warrant. If you are arrested, the contents of your phone may be scrutinized by the police, although their constitutional power to do so remains unsettled. In addition, it is possible that courts may approve the seizure of a camera in some circumstances if police have a reasonable, good-faith belief that it contains evidence of a crime by someone other than the police themselves (it is unsettled whether they still need a warrant to view them).
4. Police may not delete your photographs or video under any circumstances.
5. Police officers may legitimately order citizens to cease activities that are truly interfering with legitimate law enforcement operations. Professional officers, however, realize that such operations are subject to public scrutiny, including by citizens photographing them.
6. Note that the right to photograph does not give you a right to break any other laws. For example, if you are trespassing to take photographs, you may still be charged with trespass.
And keep smiling!!