You haven't really given any reasons as to why the kids might be better off with either you or their Dad. This seems to be about your feelings - you want them with you. As hard as it is, you need to set your feelings to one side. This isn't about you. It's about them. So, what would be best for them?
Things to think about:
- what do the kids want? Even your daughter - no, she isn't old enough to make her own decisions yet, but she is absolutely old enough to have a view, and both children need to know that they are being listened to, even if the decision doesn't go their way.
- who is best positioned to meet their emotional, physical, and financial needs? Who are they closest to? Who provides the practical support with things like homework?
- how will things like childcare work? Is either parent around more? Are the practical steps in place to ensure that whoever they live with can practically manage things?
- how will contact with the other parent work, irrespective of where they're living?
- what is the set up on bedrooms and suchlike in each house? Is either house better equipped to support their needs? Where do they have the best lifestyle?
- are there any implications for schooling from a move of house? What would that mean for them?
- are there other things that could be changed, that would help address whatever is driving the suggestion of a change of house? Are they having to carry lots of stuff to their dad's, so should they have more belongings spread between the houses?
- is it really such a big deal? You say the kids are often spending 4-5 nights with their Dad anyway - so, in practical terms, they're pretty much already living with him.
There are a host of things to think about. But you do need to come at those from the perspective of what is right for your kids - simply saying that you feel your daughter needs to be with her mum may be understandable, but it's not a compelling argument that this is best for them.
Mediation - if done well, with a good mediator - is a good way of working through those considerations. Working together constructively, to find what is best for your kids, is always the right target to aim for. Going in with a view that, no matter what, they aren't going to live with their Dad will mean that mediation isn't effective. Far better to listen to what is happening in the kids' heads, think it through, and determine what is right for them.
If you are worried about finances, take legal advice. If you're already divorced, and have a finalised financial order in place, there is no risk that he could take the house. If you weren't married, the same is true. If the divorce is ongoing, then there certainly could be a dispute about the financial settlement. You would need legal advice. But a decision about what is right for the kids should never be based on whether or not you get the house, anyway. So try to separate those issues in your mind.
It's really hard having these sort of discussions, and it can be scary. But it's the right thing to do to come back to the residence arrangements from time to time and ask if they are still right for the kids. Hopefully, both of you will be able to approach this from that perspective.