If people don't KNOW you're TTC you won't be judged, if they do know and they're nice people, they still won't judge.
Many mums (me included) weren't actually actively TTC and found out we were pregnant, counted back and realised we'd had a proper binge a few weeks before when we were probably pregnant at the time. I wouldn't judge anyone for that. Though there's an obvious sense of guilt involved.
TTC can be a loooong emotional process, and in my opinion, if you're not pregnant, you can give yourself a break occassionally, whether that means you have a drink or just stop trying for a while.
Its between you and your conscience. If you want to enjoy a drink, but are actively TTC and therefore very conscious of what's happening in your body, and would feel awful if you DID have a drink, why not take a small break in TTC?
Having never been in your situation my guilt has been in hindsight, and its very real, but if you're not pregnant you shouldn't feel bad about having a few more than one drink.