No, it's not on. So 3 of you are single and one is married? I would go beserk if this were my dh. He's in the company of 2 single women and he chooses to go back to one of their houses? Not acceptable.
Turn the tables - If you were a married woman with a single friend and you met 2 single blokes, would you go back to their house? I wouldn't because I know dh wouldn't like that at all, it's disrespectful.
If the guy's wife knew you and he had phoned her to make sure it was ok, then fair enough. But I think it's a fairly safe bet to say that if she knew, she wouldn't be too happy about it.
What you should be asking is, if it's all innocent, does his wife know where he is?