Pickles, do NOT let him take your DD.
This is his attempt at a power trip. If he genuinely wanted to see your DD, he would make a reasonable arrangement. He hasn't done this: instead, he is trying to bully you.
It is not reasonable for such a young baby to be left alone with someone she has not seen for 3 weeks. He is entitled to contact, but he is not entitled to arrange this just to suit himself: your DD's interests and welfare are what's important, and this arrangement is not in her best interests.
If possible, get up and go out tomorrow early. When you are already out, text him to say something like: "I am out today and I will not be in at all. Don't come". That way, he can save himself a journey, and if he comes anyway, then he has only himself to blame for his wasted journey.
Also write in your text something like this: "I will be in touch when I have seen the solicitor. I am happy for you to see DD regularly, but this needs to be properly arranged, to ensure her wellbeing".
Don't text tonight, and don't text before you have left the house, just in case he is angry enough to set off immediately.
Don't come home alone tomorrow either.
I don't mean to sound like a scaremonger, but angry bullies who aren't getting their own way can turn nasty, and you do not want to take chances.
He does NOT have the right to take your DD. If he comes to your house when you are there, and tries to take her, or threatens you, CALL THE POLICE.
Get yourself to that solicitor asap so you can sort this 'officially' :)
There is some more info about arranging contact here