I live 150 miles away from SD, my OH goes every Friday and picks her up, and takes her back every Sunday. We have her for 6 weeks holiday of the year. The point is, to be part of dads life, which she is. The point of living (in that part) with me is because dads life involves living with me. His ex didn't want any of these things for all the reasons you state, but mum doesn't own their child. Their child not her child. His ex tried to refuse me being present when she comes over, and was told no. I deeply regret we complied with her at first on not seeing certain friends, as she now is upset she has only just become friends again with these close childhood friends.
I understand about the expense, it is very expensive it rules our life and what we do, we have never had a non uk holiday together, for 8 hours a fortnight OH is driving. But SD is happy, and that's all I care about.
I'm not trying to be dictating to you what your family should be doing as that's none of my business as all families are different. I am just saying what we have was against mums wishes, but was granted by the court, and is wonderful for our family. So all I am saying is that if the reason you have this arrangement is it's what's best for your family right now that's cool. But if it's what happens as you don't think a court would grant you more, you are incorrect to think this kind of thing is granted as it is.
For me personally, I didn't feel that I could be with my partner if I wasn't an accepted part of his family, as his child and he are a unit. The idea of his ex dictating what happened in my life was unacceptable to me. It was important that as a sign of respect for me, and also his child, he insist on us being a part of each other's lives, as we both are of his. I personally would feel massive alarm bells at the fact he is prepared to pander to an ex and not just say no, I am taking my child out for the day, with my partner, as is my right as their dad.
I get that in the absence of a court order, and sometimes with one tbh you are under the thumb of her potentially refusing, and so sometimes you have to suck things up for the sake of not having contact withdrawn. We have done that (the not seeing friends thing I regret). But for me personally, I would look to have an agreement which doesn't allow her control like this which gives their child a damaging impression of mums place in the world. Even if it was a court order that just said he could have one day every six months he take his child out for the day, and you can decide if you go or not, in control of your own life. I wouldn't personally have anything to do with going to someone's house to be judged by some random woman who is nothing to do with my life, and I personally think the child has rights which are being ignored there and will upset them in the future.
I completely get though that life isn't as easy as just paying out to have what you want and so if it's the case you simply can't fund any of this then you are both in suck it up territory and so you should decide what you feel is best for all of you. For me personally I wouldn't feel it was healthy for me or the child but as I say every family is different and I'm not you or yours :-) I hope if you go you have a nice time.