You haven't let your son down, because you realised in time what you were about to do and didn't. You showed great awareness.
PND is stereotyped as not loving the baby, wishing s/he hadn't been born, but honestly that's cobblers. PND is a chemical imbalance in your brain, it's your hormones all over the place. And how that shows itself is different for everyone. I'd also suggest that your brain isn't coping with the relief of finally having a child. You've had that stress of wanting a baby for so long, wondering if it would ever happen, and now the stress is over (for now - you may go through it again if you want another child), your brain hasn't got to grips with the new situation. So it's everything you ever wanted, but it's also overwhelming.
Please don't try to find some sort of peace or escape in another man's bed. Go home, cuddle your baby and your DH, say a silent prayer that you woke up in time, and make an appointment with the doctor asap. Your brain is scrambled, and there's absolutely nothing wrong, and no sense of failure, in asking for help to unscramble it. And then you'll be in a state to enjoy the family you've created, knowing that you were strong enough not to throw it away.
(I wonder - was part of your thinking that if you slept with this man you'd wreck things so totally you'd have to walk away, that you'd have an escape route?)