@DrowningInSellotape "...does anyone have any advice?"
My only advice is that we all only get one life. If you end up living yours with a man who makes you very unhappy, for whatever reason, you are wasting it.
If leaving is best for you, do it. If it is best for the kids, then you can offer them a home with you.
Are they his kids, you said "... my youngest Dc (age 18)." Just make sure your kids know they also do not need to stay.
"I would be seen as the awful bitch who left the ill man, I can't think what his family would say or mine for that matter."
You can tell them you have fallen out of love with each other, and despite his being ill, he is still a person, and so are you, and life is too short to live in a marriage that is not working, for either of you.
He may choose to paint himself as the victim, and you may well wish to tell people how horrible he had become. However, I think the best appraoch is just that neither of you really loves the other anymore. If he loves you, he will be willing to change. He cannot change being ill but he could change how he behaves towards you.
"He often says he wants to go off and live on his own anyway, I wish he would." So wqhen he says that ask if he is serious. If he is how can you do this, together, and tell family together?
"Especially the youngest who has aspergers and anxiety and is often saying "You won't get a divorce will you?"
I know some people stay married for the kids but she is an adult. Could you and your husband talk together about the best way to manage this. Could you convert the home into two maisonette and have one each so you were not living together but the 'home'was there for your dd. Maybe a bad idea, just thinking! Not been in your shoes but sending huge sympathy to you. 