I'd say that how you're feeling now is completely normal. But I'd use it to make sure you don't drift apart. Remember why you love each other in the first place, have little chats about your concerns and make sure you both keep making the effort to do special things for each other. Rather than turn concerns into big issues or arguments, use them positively to get the reassurance and love you need.
(And however hard it is at first, remember that even though it will seem like every second has to revolve around the baby - it doesn't have to! New babies sleep a lot! And even though you may not want to be jumping into bed with him, a cuddle, massage etc can work wonders!)
My situation was very different as dh's affair happened as a result of many things, not just lack of time and focus on each other. The counselling has showed a number of issues that affected him a lot more than he even realised himself. (Not making any excuses here btw!)
But anyway, looking back to when ds was born, the little things we'd do for each other mattered so much. Running nice baths, making nice meals for each other, little gifts or text messages saying how much we loved each other etc etc really made me feel special.
You need to be confident about yourself too. Your partner will not look elsewhere when he has everything he needs right there in front of him. As long as you believe that, he will too.
I had so many confidence issues after I first had ds and now I look at myself and am so so so happy with what I see. I've changed for the better in so many ways. In just took me a while to realise it. And my beautiful baby is the icing on the cake.
Don't worry about feeling this way either. But also try not to dwell on it. Use it in a positive way to make sure you don't make the mistakes that you've seen others make. The fact that you're even thinking of these things makes me think you will be absolutely fine