Sorry you're going through this, OP.
Assuming the baby is his, then if you're going to continue to have a relationship with this man, I think you have to accept the fact that he has another child that he's responsible for.
Now, he comes with another child in tow, whether the mother wants him to be involved with him at the moment or not. If you stay with him, you're potentially taking on the responsibility of this other child long-term, including the financial burden that comes with that.
Even if the other mother decides not to have him involved at the moment, anything could happen to make her change her mind about approaching the CSA for child maintenance (not least when she realises how expensive raising kids can be..).
From what you've said, it seems that your husband pressured you into having a termination because he felt he couldn't afford to raise a child and then tied himself in financially to raising a child with another woman. I wouldn't be sticking around under those circumstances personally, but if you do, you'll need to accept this.