You have most of your children's time.
He has very little time with his.
You choose to include him in their lives, otherwise you wouldn't spend much time together.
He chooses to keep his small time with his daughter focussed on her.
And not introducing a partner to a young child may well be wise.
That's his firm decision about his relationship with her.
All this is fine.
It may be her mother is using their child to control him - finding ways to imply that if you become involved she'll see to it he'll see less less of her. This happened to a relative.
In which case he's coping and compromising to keep some relationship with his own child.
Maybe his behaviour stems from fear of what she might do.
But.
His reaction to you expressing your feelings about it all is NOT.
The walkaway, silent treatment? Are you walking on eggshells around him? Does he do this at other times, for other reasons?
That's out of the coercive controllers handbook.
And it's a forecast of how things may get worse later on.
Crucial question: Are you sure this is what you're happy for your children, male or female, to be observing?
Watching how a relationship happens, as in learning that this is how men treat women?