If you ask him to go expect crying, begging, pleading, calls of you being a bad mummy, threats to take your dd away. Prepare yourself for this nonsense now. It is because his meal ticket is going and he is going to have to stand on his own two feet.
The affair nonsense is either because he is having an affair, or to control what you are doing by accusing you, making you stop being friends with your male friend, thus isolating you further. He no doubt may worry he is going to lose his meal ticket. He may believe that women in ugly clothes will stay and support him as no man will be interested.
Over the summer did his accusations lead to you modifying your behaviour? Changing what you wore? Stopping you see your friends etc?
His life consists of: watching the telly, with heating on, food in his tummy, a free roof over his head, able to play with the kids when He fancies, have a bit of a strop if it's not to his liking, smoking pot when He wants. Wifey deals with all the boring stuff, employment, bills, appointments, toilet cleaning.
A good dad, bollocks is he. He is teaching you dd that her job in life is to slave away for 'her man'. While 'her man' rests, relaxes, indulges and enjoys life's pleasures. Good dad my arse.
You can do better op, but I promise you you won't ditch him without him clinging onto you with promises to change etc... You need to get prepared. Good luck op, your world should be better than this.