I'd be upset too, you're not stupid for your feelings! Don't talk yourself out of them!
I think you're reacting to being a close and social family and his refusal to engage. I think his personality chatty/social didn't match his behavior not chatty to you or your son. I would feel this and respond to it as well.
Also first meeting I'd be really grateful to be able to stay, maybe a present and make an effort. He did the opposite and isolated.
I'd worry that if they got serious and moved in together they'd not visit because she's not a family person. Don't get me wrong some people don't have strong attachments to family, and that's ok but what you've got is someone with a different dynamic and I'm the future they might not see you much because she wants her to herself.
That's a catastrophising response but a realistic one to when taken to extreme (and happened to my friend who is so close) it's only after they broken up she realized how much she was isolated by him and how she wanted someone to join in, not separate.
So I get that underlying fear of losing her.
Maybe he was just nervous and he's not done it, and hes not close to his family like that, before but I completely understand your reaction and hopefully it'll improve next time and future times. :)