This is awful OP, such a breach of your trust. Ignore the poster who has got no idea about the law around trespass, etc - giving a key to a neighbour with the instructions it is for emergency use only does not mean they have free reign to go into your house anytime they like, do whatever they like in your home or give your keys to anyone else they think might like them and there be no legal repercussions (even if purely civil court rather than criminal charges). If you hire a car for Monday - Friday, but the hire car company don't collect it til the following Monday, it doesn't mean you can use it over the weekend too! You haven't paid to use it then (and therefore unlikely to be insured etc too).
My husband had a neighbour when we first got together who had a key for his house and he hers. She used to let herself in to borrow videos (to be fair he had said she could borrow them but not sure he totally meant go in specifically to treat it like a lending library) - she'd also do his washing up for him because he left it like a shithole. I still can't believe he used to boast about it and not be embarrassed. Anyway, after I started spending more time there I left my DVD player and the half a dozen or so DVD's I had there. The neighbour helped herself to a brand new dvd (still in its cellophane wrapper) and it was only when I got my husband (then BF) to call her to ask if she had it that she said she had and hadn't realised he had a DVD player too as she thought only she did. She never even apologised - despite it being obvious that we hadn't even watched it yet (I think we were planning to watch it at the weekend if I remember correctly). She was an all round odd woman though and the whole set up/agreement they had was weird too! Even she wouldn't have considered using his house to let her family/friends sleep in when he wasn't there!
It's a very long way for your sister, mum and BIL to travel to check on the house for you, especially as at that time they are likely to be in your neighbours house (I'd presume they are only going in to sleep there... at least I hope so). I agree with the idea of using a local Facebook group - maybe even DM someone you have seen regularly post if you don't want your neighbour to see. Find a busybody and they will LOVE to get involved!
I completely understand how you're feeling. If they'd asked in advance you might have agreed (for a small cost to cover expenses) and you'd have stripped the bed (leaving them fresh linen or asking them to bring sleeping bags), you'd have moved anything personal, private, fragile or valuable out of sight/somewhere safe, etc. Or you'd have said no and taken your key back in advance (and taken any precautions you felt necessary to ensure they hadn't made copies). Either way, you'd have felt far more in control of the situation - as it stands you feel like they'll be judging your house, they'll be looking through your things, they might even steal stuff (what if you keep some emergency cash at home for example). There's also costs - what about if they left the house at 9am to go to your neighbours, but put the heating up to 20°C before they left so the house was warm enough for when they got back - and then don't go back til midnight. That's an expensive day of heating an empty house! So many things they could accidentally do because they aren't as cautious or conscientious as they'd be at home.
I hope you find out the truth. Good luck and please try to enjoy your holiday.