Hey - you HAVE put up a fight for your son, by going ahead with your pregnancy in such difficult circumstances in the first place!
What do you mean when you say "He doesn't feel the way I do"? - do you mean he doen't feel so much about your baby, or he doesn't feel he's still in love with you? Does he know that you still have such strong feelings for him?
I think at the end of the day, his relationship is going to end, because he won't be able to keep this a secret from her for ever. When your son is still small, that's one thing, but once he grows and wants to know who his father is, you're going to have to be able to tell him something, and "Daddy doesn't want his wife to know about you" just won't cut it. Your child deserves more than that, and so in your shoes I would ask him what he thinks is a reasonable course of action. If he tells his wife now, she might forgive him. If she finds out, or he tells her much much later, then it's hard to imagine she could forgive him.
You say you don't want to build your life on their misery, but this goes beyond your romatic involvement with him now that there's a baby involved. On the one hand, as you say, he already has a home life with her. OTOH, he compromised that by sleeping with you again, and so really, it's his responsibility to face up to now.
Rambling a bit but I feel for you so much in this situation, and it's wrong for you to be worrying about what you've done when in fact, he was the one with so much to lose.