He sounds like an office sleaze.
Is he senior to them?
If so he is the sad inappropriate office sleaze.
You can bet they are talking about him behind his back.
In my late 50's and I remember the type well.
Stop arguing about it.
Stop doing anything that makes his life easier.
Focus completely going forward on yourself and making your life better.
If returning to work isn't an option or unnecessary financially, definitely look into evening classes etc to expand your social life.
Make yourself your biggest project and priority.
I am in Italy with my husband and two of my 4 children at the moment and I wouldn't tolerate this behaviour.
We both like to wander off on our own, both having an endless fascination for Italian churches and the stunning architecture although you need to time your wandering well in the heat.
Don't stay lonely is my message.
He's selfish, vain, and won't change.
You need to invest in yourself.
Post away when you feel lonely.