It sounds like your husband is really struggling & was a bit naive to not have expected that anyone who marries a person with children, can find themselves in the situation of those children living with them full time.
regarding your ex partner. Is he paying the appropriate amount, considering he offers no overnights? Is your husband financially or emotionally overwhelmed, or both?
do the two of you have any breaks from all the children both shared and just yours, eg date nights, occasional night away if you have family who can help with childcare?
is your (not shared) childrens relationship with your DH good, are there ongoing problems? Are they respectful of DH?
are your first children coping with their father dropping out of their life?
This must be absolutely awful for them. That’s a massive rejection for their own dad to just not bother & must be very hurtful.
I know cost is often a major factor but is there any chance for family therapy & separate couples therapy?
was your husband just venting or did he mean it?
I know I would be overwhelmed and very stressed if my husbands kids came to live full time, they’re very full on & seem quite a few notches more demanding than any other children I’ve ever met but I did accept when I got married that it’s a very real possibility, ANYTHING can happen in the future & your DH doesn’t seem prepared for this very real concept.
11, 10 & 8 are hard ages (I found) there’s endless noise, fighting, it’s not easy. I would be a woman on the edge with 5 kids, you are doing well to manage this plus DH having a wobble. I think for a lot of step parents something does switch after the biological children arrive. They THINK they love the steps until ‘the real thing’ is in their arms. And although their own children are seen through rose tints the step kids little annoyances are magnified hugely. It’s not fair though & your DH needs to try & imagine for himself, if you split & his childrens new step father had this sort of resentment towards them.
you’re in a rough situation. Obviously you aren’t going to start giving your children to your ex in the next 2 years as they become teens, that’s ridiculous & I’d like to think this is more DH is venting & talking rubbish, than actually means it.