I used to have a real social anxiety problem--though that's probably not your issue--so I know how it feels. The thing to keep in mind is that everyone has insecurities and issues (not just you), you'll probably remember your screw-ups for longer and as more serious than anyone else remembers them, and even if you're not particularly well-liked, there's always people who'll be on your wavelength. Some places will fit you more than others.
If you don't like confrontations, just learn a few simple ways to politely leave the conversation, or just don't let them involve you in it personally. It's not for everyone.