How old is the child? If a toddler/small primary school age, there is something to be said for them to be on home soil, with their toys, their comfort items etc.
But if the child is older, surely they can build a better relationship on their own, rather than in the ex's house. Presumably she is around, and child may well defer to her, wait for her perimission etc. And if a teenager, they may well hibernate in their room rather than spending time with Dad.
I think in the interest of having time being the parent to his child, he should have the child in a neutral territory - perhaps yours is too far away, but a B&B, different city etc.
And if you guys are in a committed relationship, why are you not part of this child's contact? Surely if you are important to your partner, he would want you to know his child and vice versa.... you've been together 7 years, I think the child might deserve to know you too.
I'd be rethinking the format for the contact with your partner... suggest child comes to his place once a year for a week, and partner takes child away for long weekend somewhere away from home.
Personally, the jealousy side is your problem to deal with, but the exclusion from his child's contact and his relationship with his child being under the ex's "supervision" would be the approach I would take if I wanted this to change. Silver lining might be he no longer stays with ex, but the reason for this shouldn't be your jealousy.