My friend is in this situation:
She left her partner and moved back into her Mum's with her two children (both have different fathers). The eldest is in my friend's care full time as her Dad is not allowed contact after social services got involved. The youngest's Dad is not allowed contact alone as he forget to feed the baby and give him his regular medication so he aranged for his Mum & Sister to take the baby each weekend (Friday eve - Sunday eve/Monday morn) and he will spend the weekend at his Mum with the baby - this happens EVERY week. My friend has said that when the baby comes back he will not settle for her, be comforted by her. She has also mentioned that baby's Dad (her ex) has threatened with taking baby off her if she causes a fuss about the arrangement. My friend never agreed to this as she feels her baby is too young to be away from her.
My friend is scared to mention anything to them incase they take him away. I mentioned that she should speak to her social worker.
Sorry for the long post but what would you do?