Hi OP,
First of all, I am so sorry for what you are passing through. In no way do you deserve to be betrayed like this. However, I do happen to agree with some of the things you said about your part to play.
In any marriage, regardless of the length of it, you need to keep working it and keeping the spark alive. If you have been upstairs a lot like you say and he has been downstairs and you have not become physical then he has found outside what he could not get at home. Theres always lots of reasons why you go off sex in a marriage and sometimes life is just too stressful to fit it in and take time out for you both as a couple so no judgements here babe! However, at the same time (take it or leave it) men need affection and need to feel appreciated from their spouses. This is just in the same way you also need to feel the same things from him.
Taking away the sex aspect for one moment please, most times men cheat because they are looking for someone to give that emotional support that they are also missing at home. Another factor is that there is temptations everywhere no matter how happily married you are. We are all humans and its difficult sometimes to avid the hot guy or girl looking at you. There will always be some women somewhere at some point who will try to get my husbands attention but when is so happy, getting a good sex life at home, emotional needs, appreciated, he will not look twice at this other women.
All this advice (so far) really does is explain why your husband did what he did. Look babe, on this website its very common for people to just say LTB I see it a lot!!! I am not saying that you should stay in an unhappy and unhealthy marriage by any means but, you should at least try first and find a way to overcome this, build trust again with your husband and start again.
I think the first thing you really need before you can even give him a second chance or attempt to rebuild is to make sure that this other women is permanently gone from his life. If he is still swaying back and forth between her and you then I would be included in that situation to take your self respect to the door! But if you can be confident enough that he has finished things and is committed to you (suddenly found his brain)! now then perhaps try counselling and fixing the things you mentioned.
It sounds to me like you both need to talk more, about your feelings, needs, wants. When I say wants and needs I mean as individuals and in your marriage! You just need to emotionally connect again. As for the physical stuff, I think thats going to take time because I would NOT jump back into bed with my husband after an affair was discovered, it would be to raw and painful. Just because, I say stay and try to work things out does not mean it should be easy for him or will be painless for you.
Another thing I think you should do is to remind him what he has been missing at home and make him feel so STUPID for betraying your trust. I think you should give yourself a right out makeover, new wardrobe, underwear, haircut (depending on what you like!), start a new hobby, become more active (gym, swimming, yoga, running) and just show him how SEXY!!!!!! you truly are. If he feels bad now he will feel worse when he remembers he has all he needed at home and he settled for what I like to call ^his 20%" (Your his 80% btw!).
Also, I had a quick skim of previous posts. Has this been going on for 3 years? I got confused by your first post and I thought perhaps 3 weeks? If it is 3 years then things are a lot worse than I first thought but it doesn't mean you can't come back from it all. I am just so shocked but at the end of the day you can only decide what you want to do. We can all give you some advice on here and tell our wee bit but its up to you and your the one living with whatever you decide.
I think maybe you need a holiday away on your own. If you have any friends you can go away with for a week I think you should do that and leave your DH with your child. I really do hope you work through this and at the very least your DH realises what a mess he has made of his own life.