Some of this comes from my own experience. It is a personal opinion- please do with it as you will.
Nearly 3 years is a long time for your relationships as adults to have gone on the back burner. Maybe he was feeling unloved, or remote from you? I know you have a very busy life, but it strikes me that all this running around and sleeping apart etc has left very few opportunities for you to actually talk to each other or even share a hug.
I know how easy it can be to erode all your time as a family by filling up weekends etc with other things, but maybe it is time to say no to visits to family etc except for maybe once a month, and to do things together, that you want to do. It sounds to me as though your whole family unit needs some down time.
It may not seem as though you will ever regain what you had before,but actually this could strengthen your relationship, if you allow it to. He was wrong, he nearly made a mistake, he hid things from you, and he has to understand that he can't make it a habit. But, if you can use this as an opportunity for talking, then you have everything to gain from it. Yes, you are pissed off with him, as you have every right to be, but if you love him as deeply as I think you do, and he by your account seems to love you equally, then all is not lost.
The woman in question sounds as she may in the same situation, with the added emotional vulnerability of the infertility to deal with, maybe she has stopped talking to her husband and started talking to yours instead. She is talking to the wrong person, he is talking to the wrong person, but they both happened to be looking for someone to talk to at the same time.
I would be tempted to: Cancel relative visit!!!! Tell family you're having a weekend in for a change. Find a babysitter for your daughter for the (entire!!!) weekend (a good friend might be best) and invite your husband over for tea on Saturday afternoon. You will have some time to yourself for some part of Saturday in which to reflect. When he comes, talk. Tell him you love him, tell him how hurt you feel. Let him talk. Don't let him fob you off with trite excuses and promises- make him talk, ask him questions if needs be to make him open up. Let the talk run itself. Try not to have a blazing row. See where you end up.