It’s frustrating but he gave up his job to be a house husband, and he has his name on the house deeds. If the situation were reversed and a woman had given up work to allow her H to work hard and earn the family money people would rightly be saying that she deserves half of everything (if not more) - regardless of what a shit job her ex says she did of the housework. The fact that you had live in help has obviously made his life easier so any settlement should reflect that, but as you referred to him as a SAHD I presume his presence at home added something to your family life, even if only the ability to travel for work while one parent was still there etc.
FWIW my DP’s ex was a SAHM and an alcoholic and he is resentful of having to give her money as he knows it goes on wine not DCs school shoes etc (he buys them everything they need, clothes, clubs, all Xmas/bday gifts from him and mum etc). However he still gave her a large sum, even though they weren’t married and she wasn’t legally entitled to it, because it was the right thing to do, on top of child maintenance (which technically he doesn’t need to pay either, as he has the DCs most of the time).
Predictably she has spent most of it down the pub and still comes to him asking for more money regularly, but he can hold his head high knowing he hasn’t shafted her, the mother of his children.
Obviously get a good solicitor and try to get yourself the best deal according to the law where you live, but if he’s entitled to anything, even if he doesn’t deserve it, then you will have to make your peace with that. No reason why you can’t make him work for it, but if taking you to court is going to cost him a lot, bear in mind that’s all money wasted too, albeit in solicitors’ pockets instead of pub landlords’.