Stay calm and leave him to it.
Clearly he wants to stir the pot, and looking like super dad is also appealing - but at the end of the day, being 'involved' in any proper way will take... ohh
here for some terrible words... EFFORT! COMMITMENT! TIME!
Certainly don't let him in your house - a flat 'no' to that, followed up by 'here's the details for you to make your own appointment so that DD can be assessed at your home' - hmm see if he actually bothers with that one.
Create the impression of 'ooh great, if you want to take responsibility for stuff that happens at YOUR home you are more than welcome to make your own arrangements' - you won't see him for dust.
Remember that a big part of this is him trying to get under your skin, so NEVER, NEVER be rattled by this in front of him. He'll give up sooner. If you give the impression that you are grateful and flattered by his interest, he'll give up even sooner!
Also speak to HV and GP and make them aware of the abusive background just in case he does come to any appointments, and ask their opinion on sending out letters - say you'd rather it be just to you, despite his request - would they support that, as it would allow you to manage the situation better.