Not a chance in hell.
Email him (so you have proof in writing) telling him that although he is welcome to see the baby at your home, you will not agree to him taking her away from you at this early stage, especially as she is exclusively breastfed and cannot currently be separated from you for any significant length of time. To do so would be distressing for the baby and potentially harmful as early weeks feeding becomes established. You are also advised that this level of contact without separation from the mother is normal and follows acceptable guidelines.
Tell him that you have taken legal advice and this is your position, and you have no intention of discussing this with him further, so if he wishes to pursue this he needs to take legal advice himself.
Add that you are shocked and saddened that he has already threatened you with giving the baby formula against your wishes, and has been aggressive and threatening to you whilst demanding that you are separated from your newborn, and appears to be unable so far to act in the best interests of your baby. You are therefore only going to agree to him spending time with the baby in your home with another person present as you are worried that he will attempt to take the baby away from you.
Send him that, and if he waves it at a solicitor they will very quickly advise him to get a grip, understand that he has NO chance of having the baby without you present and certainly not for a full day for a VERY long time and if he persists in acting threateningly towards you, he won't even get that.
Don't for a MINUTE think he will get anywhere - your absolute best bet is that he DOES take legal advice, because he's on a hiding to nothing.
If this is the way he's behaving, take a hard line. Baby goes nowhere without you. Contact in your home with someone else there too. He's clearly not able (yet) to put the baby first and seems to see her as a possession. Until he snaps out of that, snarl back and make sure you make it clear that no, he will NOT be dictating how you mother your child for those vital early months.