Thing is, if you refuse to pay him anything at all, you can kiss the relationship goodbye now. That's not how healthy happy couples treat each other.
Okay so you might have been happy with beans on toast, but would you have been happy to feed your child that for every meal? If he paid for your child to be fed well, adequately clothed and housed for a substantial amount of time (when your ex wasn't paying) then it's fair for you to pay him back.
If he wants you to pay him back for what he has spent on YOU, then you do that from your own income. Child maintenance is not income. That is for your child, and your child only.
Both of you need to sit down and figure out what he has spent on your previous child. That is all you owe him (from the child maintenance).
Then, because you are uncertain about the relationship and don't want to be left with nothing to your name if you were to split, you pay him it back bit by bit each month.
This way you aren't refusing to pay him back, and you aren't leaving yourself short should he leave you whilst your financially vulnerable.
That's what I'd do anyway, and I wouldn't budge on it. As another poster said, put your children first.