Bugsy, my heart is crying out for you although, at this moment in time, the last thing you need is someone's sympathy but I know exactly how you feel as I have been there, although circumstances slightly different - it was me that had the affair but ex-h couldn't forgive me after having 3rd child and then left (although stayed in the house for 9 months, although we were not together).
Reading your posting about the text message just makes me feel so hurt for you, it is such a horrible thing to read as I found out when ex-H started seeing his girlfriend and kept sending me text messages by mistake!!!
He keeps asking you to hug him and then wonders why you give him the cold shoulder - because of the bl**dy way he has treated you. How can he expect you to "comfort him" when it is him that is making you back off in the first place.
It sounds to me like he is very confused (as was my ex-H - that is why he took so long to actually move out of the house) and doesn't know what he wants, or perhaps he is "weak" and is scared of making that final break.
You are strong, and you will get through this.
I know exactly what you are going through having been through exactly this same thing this exact time last year. I had so many friends saying I should "kick him out" but at the end of the day I was scared of taking that route. Although we were not together, at least he was still present in the family home although he kept going out at weekends, not coming home, staying with his girlfriend but at that time I didn't have the courage/strength to kick him out.
It was only when I went on holiday, having left the baby at home with him, and then finding out he had spent both weekends with MY baby and his girlfriend that gave me the courage to "kick him out". It did take me a while to get over it, but I can now say I certainly am.
I know I found it hard that my husband kept kicking me when I was down (phrase - not literally) but, at the end of the day, I let him.
You will find the strength to get through this, as I did. One day, and no one can tell you when that day will be, you will wake up and realise you are worth more than this piece of sh*t and you are a worthy, loving, kind, intelligent woman and mother.
My thoughts are with you.