Previously posted about this in AIBU but it was suggested I posted here too, so I'll start from the beginning.
Ex-bf and I met at secondry school. We were both excluded from school around the time we first got together and were using illegal substances, I was living with him at the time after a fallout with my mum. During that time we had a child together who was sadly stillborn. At 16 I sought help and ended up in a private fostering arrangement, went back to school and stopped taking the drugs. I have a brilliant relationship with my foster mum and still live with her now. Ex-bf and I were on/off for a year or so after that, during which I had a brief relapse but stopped when I found out I was pregnant with our daughter- the two of us haven't been together since but have remained friends, and I've been completely clean. DD is now three and I would never do anything to compromise her wellbeing.
Until recently ex-bf hasn't really seen DD on a regular basis- for about 4 months he's been having every other weekend with her, those weekends he picks her up from preschool on Friday afternoon and brings her back on Sunday afternoon. This weekend was one of his weekends.
On Friday, one of my best friends who works at DD's preschool told me that when ex-bf picked DD up that day, he asked to have a word with her and said he had reason to believe I'm using again and is afraid for DD's welbeing. He claims to have seen me under influence herself- that is absolute rubbish. This weekend he's also texted me to say he's going to push for 50/50 custody as he doesn't think I can look after DD full time, he's also texted my foster mum asking if she's got any reason to believe I might have started using again.
He dropped DD home today 45 minutes late and said she was anxious about coming home and seemed very distressed- both my mum and I thought she was absolutely fine. According to DD, she had a good weekend but Daddy asked her some funny questions, like whether she'd seen me acting strangely or scarily, and how many times I'd been out after her bedtime recently 
I'm absolutely terrified he's going to try and turn my past against me :( I'm seeing a solictor next week and having a hair sample taken to proove I haven't been taking anything, but in the mean time any advice/hand holding would be much appreciated.