I would feel the same as you, but wouldn't let him in the house at that time.
Your DD has wasted two years on this waste of space? 
Is he polite to you? Does he engage in chit chat, or are they off on their own the minute he walks through the door?
I think I'd say "Oh, hi, come on through & have a glass of wine/cup of tea"
Then I'd ask him how his kids are, take an uncomfortable interest in them & his situation...
How old are they?
What are their names?
Which school do they attend?
Do they enjoy school?
Do they have any hobbies?
Who do they take after..you or their mums?
OOh show us some photos!
Followed by selection from the following...
It must be so difficult having to live with your ex when you're separated...
How long is it you've been separated now?
So what are your plans....is the house on the market?
Do you get on with the two mothers of your children?
Be really nosey, all with a friendly, smiley disposition. He will squirm.
If your daughter kicks off, I'd point out those are completely normal questions to ask, and it's weird not to know all that stuff when you've been "boyfriend & girlfriend" for 2 years!
Basically, I'd want him to realise that coming to your house is not worth the hassle of you being on his case, making him feel uncomfortable.