TLV - for what it's worth I totally agree that you shouldn't invite him to do the family things anymore.
I say this from bitter experience. My H left last December saying he needed 'space'. For months I tried to 'show him what he was missing'. I let him come and go as he pleased, he did all the lovely family things with me and ds, slept with me when he wanted... then he would leave again as 'his head was a mess, he was confused, he needed space...'.
Much later I discovered he had been having an affair. He was literally living a double life.
(He too never went out much so I did not suspect a thing. He was doing things like meeting this girl at 8am in the morning, having sex with her in his car, then dropping her at work!! Then the same at lunch times or after work. )
Me allowing him to still be a big part of our lives meant he was literally having his cake and eating it. The lovely family stuff, doting wife, me at his beck and call, lots of time with ds, making his favourite meals etc etc... but then his girlfriend in the background when he felt like it.
It's taken me (many) months to realise my mistakes. Everyone gave me great advice but I just didn't listen. I loved him too much and wanted to spend time with him - under any circumstances.
But as much as I still love him and miss him now, I have realised that you don't treat people you love like that. You don't just up and leave, no matter how difficult your relationship is. You have a child and a responsibility as a parent to speak up and deal with the issues like a responsible adult.
These men have not done that. They are selfish and weak.
He will realise what he's done. And he will want to come back one day. I just hope that when that day comes, you have the strength and confidence to tell him where to go. (And you will get to that point - I promise.)
Nobody deserves to be treated like this. And that awful feeling of loss, rejection, and inevitably blaming yourself for taking them for granted... it's just soul destroying. He is in the wrong here. Not you. It will get better. Keep posting.