I think you absolutely have to go with your gut, on this. Only you know your H. We don't.
If my first H had done this, categorically I know that he would have been chancing his arm, hoping to get invited in, that the wine lead to more wine, and that that would lead to sex. He had form for this type of thing.
If my current DH did the same, I know beyond doubt, that it would be innocent. Perhaps the friend said she was bored - and out of some sense of duty, he would have said "come to the pub on Saturday and watch the footy"
It has taken me many years, to get over the betrayals of H no 1. Poor DH has ridden it out with me, and shown me over the years, that some men don't ever cheat. I've snooped on his phone, devices, laptop (early days), and never, ever found anything of concern. Terrible I know, but I was wounded and in self protection mode. Anyway, that was many moons ago!
Anyway, point is, what type of man is he? Only you know.
I can totally see why you aren't keen to lose your home, upset your job situation etc, more so, because you only see your H at the weekend anyway. I would suggest throwing yourself in to activities and building a life for yourself that doesn't include him. If you do split up, only you will know when the time is right. Nobody can rush you in to that. One day something may just click.
I would be very interested to hear your friends take on the situation. Extracting that may be easier said than done. She'll want to minimise it, for your sake, she won't want to hurt you. I think I'd be tempted (with H no 1), to get a few wines under my (and her) belt, and tell her you're having doubts about H, and ask her to please be honest with you about how it played out.
Last but not least - he works and sleeps away Mon-Fri? Have you any clue what goes on in that part of his life? That's a cheaters dream scenario, surely? Not saying that he is cheating, but the opportunity is there.
You don't say if you are still intimate with him?