I am preg with no.4 - due in April - and was in an almost similar situation to you except DH didn't want a 4th and I had accepted 3 was my lot and plenty enough work, thank you very much and was happy with 3.
However, after going to the Doc and discussing the alternatives, getting the leaflets, taking time to think, we are having a 4th baby.
Our children are 4, 3 and 1.
You are happily married, as are we and this sort of made the decision for us iyswim? Initially DH wasn't interested but once he took a week or so to think about things, he quickly came round and is very attentive/supportive/excited.
We had sex once, we were a little tipsy, DH... ahem... withdrew and I still fell preg. Coupling that with the fact we are happy and stable and apart from the house being a bit 'cosy' once this one arrives, we sort of felt that there was no reason not to go ahead. I am not suggesting it is that simple for you of course.
You must tell him asap as otherwise you will feel terribly alone and sad. You need to know how he feels.
Not sure if my ramblings are of any use, but didn't want to leave this unanswered as I have recently been in a similar situation. x