HI POL, I've just read through this whole thread. I've been in your shoes (nearly) with neighbours of ours, I could have written your post 5 years ago.
I also live in an area where I don't have to lock the door. Kids wander in and out and ours is the house where lots of neighbours pop in for a cuppa and I love that.
However, one couple totally took the piss a few years ago. I and the other DW were SAHMs at the same time but she really didn't enjoy being on her own with her kids (used to call her older DD "a little bitch") so she used to come to my house almost every day. "I saw the car so I presumed you were in." I like her but was getting very sick of constantly having someone in my house but didn't know what to do to get rid of her. I'd start making dinner thinking she'd take the hint and go, but she'd take out a vegetable peeler and start helping. My mam would call for a cuppa and I'd think she'd go home to let us have a chat, but no, she'd pull up a chair. Her phone would ring in the evening and it would be her DH asking where she was. "I'm in lettinggo's house." Two minutes later, the bell would ring and in HE'd come too. I got hard enough eventually to be able to say "I'm going to have to throw you out now cos we're ready to eat."
I just didn't know how to extract myself from the situation. The final straw for me came just before my DS started school. By then, she'd gone back to work part-time and often asked me to do things for her because she was so so busy now and I had all the time in the world. She's come home from work in the afternoon with her kids and ask if I could mind them for a couple of hours so she could get the shopping done/ go to the post office...I resented being asked to do stuff but never said no cos I'm a non-confrontational chicken. Then she had a party in her house and had hired plates etc which needed to be returned. She asked me to return them to a place the far side of the city. I had just been talking about how awful I was feeling (I had a vomiting bug) and when she asked, I initially reluctantly said ok, then when I heard where I had to go with them, I said no, I didn't know where it was. She said "Don't worry, I'll set up the GPS for you."
Another neighbour called over, remarked how awful I looked and said go to bed, I'll take DS over to my house. I told her I had to return the plates for PissTaker and NiceNeighbour said "Tell her to fuck off and return them herself. What a bloody cheek"
So I did. Well, I didn't say fuck off but I said I wasn't feeling good. And that was the day I grew balls. When she'd stick her head in the door and shout out "Only me, I brought biscuits, fancy a coffee?" (with two children in tow) I learned to say "Sounds lovely but can't do it today because X is calling later and I've loads to get done before then/ I have to go to X and need to get sorted first/ never got a wink of sleep last night so NiceNeighbour is going to mind DS so I can get some sleep etc. I'll give you a shout later in the week if you week. What about Thurs?"
I really feel for you because you sound so like me. I hate to be rude to other people but as other posters have said, these people will bleed you dry and not worry one bit about it so you need to act differently to how you normally would. You need to extract yourselves from the situation now. When you were there last night and ODH came when you wanted to put your kids to bed, you need to explicitly say "I'll need to turf you out now, ODH, cos WE have to get these kids to bed. It's a two-man job in this house! We'll catch up with you next weekend." Just be very very direct so there's no wiggle room.