There are two issues here. Firstly, having had an affair does not define one as a parent, regardless of what posters on here would have you believe. If you have been the main carer then this is the normal that your children know, and the fact that you have chosen to leave the marriage should not change that just because your ex has been subjected to your infidelity.
However, staying in the family home is not the be all and end all, and if you want an exit from the marriage then you might do best to seek to sell the house and move on with your children and work out an amicable access arrangement with your ex. If you co-own the house then under no circumstances should you leave without the children or without first having come to an arrangement with regard to what will happen with the equity etc. You are no less entitled to a fair share of the equity than you would be if you were divorcing for other reasons, and this includes potentially being entitled to a higher percentage if you have given up a career to bring up children and will need to find somewhere to live which includes you all.
Secondly however, as you were the one who has had the affair, your husband is well within his rights to want to file for divorce on that basis and to want you to be the one who leaves, especially considering you are still in a relationship with the OM and are planning a new life with him.
Having had the affair means you have to accept responsibility for your part in the marriage breakdown. Even if the marriage was unbearable before that, the fact is that you had the affair and are now planning a new life, not because you want out of your marriage and want to be alone, but because you want to build a new life with the man you left your marriage for, and your husband's attitude that you shouldn't just be facilitated to do that in what was the family home isn't unreasonable.
And I disagree with the advice to remove half of your savings etc. Do bear in mind that if you go through solicitors everything will be taken into account, so even if you take half the savings now these will still be taken into account when the consent order is drafted and you will be required to put them back for the purposes of dividing the finances.