I am in the same boat.
I am now about 10 weeks and conceived on holiday. I feel a bit daft because we were 'trying', but for various reasons I was convinced that that month was not the month. I had just come off the pill a month previously and had had a very short (one day) period, followed by cramping most of the way through the month. I was convinced my cycle was all over the place and I wasn't ovulating again yet.
Anyway, we were on holiday and drank wine with dinner every night (almost). One night we went to a swanky restaurant and had the tasting menu, with wine for every course...
Like you, when I was late and did a test, I panicked about what I had done to the poor little thing. All the books said "don't worry if you've had one boozy night out" and I'd had 'a few' (ahem) nights of putting away half a bottle of wine a time.
I found some medical research online which I found very comforting. In line with what others have said, it had basically found that alcohol in the early stages (i.e. before a missed period) was unlikely to cause an issue. If drinking was really extreme, there seemed to be a slightly increased risk of miscarriage (although the report was at pains to stress that, as miscarriage is unfortunately so common at a very early stage, this did not mean that women should blame themselves if it happened), but it seemed to be unlikely that the pregancy would continue but the fetus would be harmed. As others have said, this was to do with there not being an established blood supply at that point.
Basically, so long as you stop when you know, that seems to be the main point.
Hope that helps. I am aware that Googling things is not the most scientific approach, but it's so frustrating when the books don't answer your particular question, and I found that information comforting. Happy to be corrected by actual medical experts!