there is a need to be careful here before saying 'no way'. What are the current arrangements for the children? how old are the children?
Please do take care with how you now proceed with things. Don't let him upset you and don't take anything he says at face value - get good legal advice and/or have a look at wikivorce.com on the internet. Courts don't generally award the residence of children to a father and his new girlfriend because they are a 'family' and the ex is now a' single mum', for example (this is something a lot of men in particular seem to believe and you will see a lot of reference on step parenting forums to the ex being 'just' a 'single mum') and giving up his job before he has secured residence of his children will not be looked upon as something reasonable to have done (unless you are an alcoholic or otherwise will struggle to parent in some way so he needs to be there for the children because you can't).
Assuming your children are quite young (under 12) and you have been a stay at home mum or work part-time and/or have done the bulk of the childcare and childcare arrangements, and that you are, to all intents and purposes entirely mentally and physically healthy (nothing to suggest you might struggle on your own with the children), it is highly unlikely an attempt to secure residence would be won in court. And a half decent solicitor will tell him this. However, that doesn't mean to say he will speak to a half decent solicitor, or follow a half decent solicitor's advice, nor does it mean he won't tell a solicitor a pack of lies about you and your married life together (which is what my ex did to get me into court) to get into court.
Things might work out a little different with older children who will essentially be allowed to live where they want, assuming your ex is as capable as you are of caring for them.
It is normal to make this kind of threat when relationships breakdown so do what you can to rise above it and please, whatever you do, don't threaten him back. You need to be particularly careful to never say anything along the lines of not letting him see the children as this will, ultimately, be used against you if he does start carrying out his threats. And few new partners - particularly affair partners - want the care of the children as it will play havoc with their relationship.
Odds are in your favour - just see it as a game you are now in for the long haul. Get good advice, have a good friend to support you and be careful what you say in distress and/or anger. It will all work out in the end.