I agree completely with TheFirstMrsDV, OP.
None of this is your fault, it is just unfortunate that you fall into a higher risk category. But so does anyone who has experienced prior mental illness, domestic abuse or violence, is an asylum seeker, an eating disorder, is in poverty or any number of prior life experiences. A lot more people than you would think fall into these categories, and everyone is very quick to criticise social services if a parent fails and they haven't picked up on any risk factors, so they have to ask the questions - they are just doing their job. The manner in which they actually do their job and ask the questions can leave an awful lot to be desired on occasion though, which can result in parents actively shying away from seeking help and engaging with professionals.
OP, whatever their life experiences, most parents will read a book (or few) about babies/parenting, will attend antenatal/NCT/breastfeeding classes/baby groups or will seek advice from friends or family about their baby. If you don't do these things, and you don't have children already, how do you learn what is normal and what to be concerned about? How do you know whether your baby not sitting up yet/not eating much/getting a funny rash/not walking or talking yet is normal or not? If you go to baby groups, you tend to see what the other babies of the same age are like, how they are developing, and what is normal or out of the ordinary in your own child in comparison. Say your baby isn't walking yet by a certain age - without using these resources how do you know when you need to see the GP and when things are just fine and you are worrying unnecessarily?
I didn't grow up in care, but rarely see my family due to distance. I was the first of my friendship group to have a baby and had never even held a baby before. I had no clue what to expect and how to perform the most basic of childcare tasks, so I went to the classes and groups and read many books and websites to help me know what to do. My HV wasn't overly concerned about us as she saw I was doing these things.
Now, I hated going to baby group. Most of the other mums were much older than me, were all married (I wasn't), had huge houses in nice areas and fancy cars (unlike me!) and appeared to have family incomes way, way higher than mine. I felt very intimidated by that and felt I stuck out like a sore thumb - hated well meaning mums with plummy accents asking 'So, what does your husband do?' and so on. But actually, once I got over myself a bit and made an effort to keep going, they turned out to be pretty nice and good to chat to about general child development stuff. They didn't look down on me at all, I'd just felt like they would. They weren't 'friends', we didn't socialise outside the group and I never saw them again after I went back to work full time. But they served a purpose for me at the time, and kept the HV off my case as they could see I was trying to 'access support' as they saw it.