While he is probably talking a load of bs aimed at upsetting you please don't assume he will only get sole residency if he can prove you are an unfit mother.
My neighbour's ex tried to get her declared as an unfit mother, it was decided she was and still is a perfectly fit mother. However her ex still convinced a CAFCAS officer and judge to give him sole residency of their two kids as he appeared able to deal with their daughters 'difficult' behaviour. He did this by threatening and intimidating the children to make them behave when the CO was around. Nowadays he punishes them for bad behavior be cancelling visits to their mother amongst other things!
My neighbours sol told her not to worry that her ex would never get SR. But the fact he was violent towards her was considered 'history' and that she should move past that. Also he had a new gf, widow with two kids, which the CO favoured. It also went against her that she stopped all contact with dad and made it extremely difficult for him to see the children while he promised he'd do the 6 hour (each way) drive every three weeks and every holiday because mum didn't have a car etc. Course that went completely out the window once he got custody!
Several years down the line everyone now admits mistakes were made and she has a very good case to get her daughter back. But she doesn't want her son left on his own so will only go back to court when she is certain she will get both children.
I don't want to scare you, and from what you've said I really doubt he would get sole residency. But please don't be complacent.
Make sure you are the one being 'reasonable' and he is the unreasonable and obstuctive party. Try not to lose it with him. Write letters regarding all dealings, contact etc to show your reasonableness and insist on written responses. You will then have documents to help back up your case. Record with an mp3 player all exchanges and document everything (no matter how trivial) so you can show you can acurately recall all and every incident.
Also CO don't like to see one parent undermining the other. Saying you are an unfit mother is likely to harm his case. So just remmeber that everytime he tries to make you feel that way and don't get het up.
Perhaps suggest mediation as a way forward. Get it in writing if he refuses. Court will want to see mediation tried anyhow. If you can come to an agreement so much the better. If not have your case ready for scrutany.
Ignore his comments. He says it to hurt you. If you really were an unfit mother he'd have called social services surely. And so would your daughter's school, your neighbours and everyone else concerned for your child. Noone has.
Stay strong. Focus on what matters, your daughter, and her welfare.
Take care of yourself
Gilly