I think you have to go with your gut feeling here motherbeyond and if you are happy then all good.
With my DD at playgroup she went to one friends house, once after nursery. But I'd met the mum, popped in for a cup of tea and was happy all was well..
Now she has started reception, I am nervous and wary as, suspicious as it sounds, I don't know the parents all that well yet. I would always ask to go with my DD to settle her in if it was a new playdate, and if I was happy to leave, then I'd leave, if I was uncomfortable then I'd stay.
The thing is, yes it is a life skill they should learn, yes they should be respectful of other peoples property, yes, they should be fine, however, with my DD I know that at just 4 years old she does not tell me everything nor still understands about personal safety even though I do tell her about it. She is just still very young for her age, and understandably, as the parent, I make the decision that she is too young to go without me going with her first.
If I was really not happy about the other parent, or their home (i.e. not very safe, or people hanging around that I was unsure about) I would suggest meeting at the park, or wherever.
I'm sorry, I know it sounds suspicious, but I do have my reasons for saying this, as I feel as her parent, I want to give her freedom, but at just 4, I am still responsible for her.
I know other children at four are more vocal and open, and it's fine for them, but my DD, I'd still be worried.
And motherbeyond, if another parent asked me to go out for tea to get to know each other it wouldn't worry me, as I can understand wanting to get to know the other parent and children first. Hope you have fun :)