DH and I have 2 adopted DDs, now 9 and 6, who are full birth siblings. Their birth mum had another baby, a boy, 3 years ago. My DH had been getting rid of all our baby equipment so I knew he wasn't expecting any more DC. But I wanted to apply, as it would have been an opportunity to have a small baby, our DDs came to us aged 1 after being in foster care.
SS turned us down because of the issues that were coming to light with DD1, which have escalated since then. It was the right decision.
But it still hurt. DH has since told me that he never really wanted another child anyway. It would have been so much better if he'd said so in the first place (although I should have realised it myself). But it would have been better if he had spoken up and not let me become emotionally invested.
I'm happy now, 3 years on, as I would be potty training now, a process I'm very glad I won't be going through again! 
So what I would say is, it's good that your OH has been honest about this. You just have to decide whether you're happy to accept what he's said. If the relationship is right for you, don't just throw it away, because you may well end up with no relationship and no second child.
But then, maybe he is just worried about you and you can talk through together what you both really want. 