he said that he 'He didn't want to put the wind up me' but that he was going to take this to the family court to get proper access arrangements
Smile sweetly and tell him to bring it on you'll be consulting a lawyer with a view to divorcing him for adultery and naming MsControlPants as co-respondent.
This should serve to put the wind up one or both of them as, if they don't get it, no doubt their colleagues will be falling over themselves explain that this means she'll be liable for a share of the court fees. It'll also serve to scupper any plans they may have for her to be rolled out as an 'independent' character witness for him in any childcare proceedings, or some such stuff of their shared fantasy that he can oust you and install her in your home in one easy movement. As if!
It could also be that the organisation they work for has some rule or other relating to 'office romances', so to speak, as their professional standing will go down pan suffer if word was to get out to the clientele that, in the vernacular, MrWoolly is a legover merchant who's been having it off with MsCP. 
It's the old adage. Give some people a little power and it goes to their heads and he no doubt fancies himself as a force to be reckoned with on the strength of his futile dealings with thugs who beat on women. Stupid twat!
With regard to lock changing, t'internet is full of helpful guides to removing and replacing lock barrels. Of course this would be a necessity if you were to lose your entry door key(s) and it may be some time before you came across the spare(s) that you carefully put somewhere safe to give to him in due course. 
It's time to play poker, woolly, and you can rest assured your dc won't be going anywhere because you've not only got the winning hand, you've also got a full pack of aces up your sleeve the combined wisdom of mumsnet dealing the cards. 