TOK I am so sorry you appear to have an insensitive partner. It is a flippant comment, made without any thought to how it might come across. I bet, if you had recorded it, and played it back to him in a few months, he would be horrified that he had said it. It sounds like one of those things you say in the middle of the night when knackered 'i hate breastfeeding' or 'you never give me any help' (which i am sure must hurt as DH does, but at 3am when he is snoring, i only remember the times he is on the x-box, or, well, snoring).
BUT, that does not stop what he said being extremely hurtful. And I do not think you are making more out of it than you should. I think you do need to broach it with him maybe immediately, as you don't want the pain to cause long term damage between you. Flippant comments are the worst as they sting so much, and can be so callous. DH is so good at them. I rarely say anything flippantly, even in anger (unless it is 3am) because I know, that once something is said, whether you mean it or you don't, it is out there and it cannot be taken back again. Even with an apology.
If you don't think he will take you seriously, or say you are making a big deal, could you write it down? Explain that you do know he loves you and the children, but that you feel as you explained in your post? And that you need reassurances that he does not regret the children.
If it is really true, what do you think you would do/say about it? Will it change things between you do you think?
[hugs]