I opted out of HV involvement for my second child after birth, my first had several HV including one who said something very upsetting to me when I was already finding things difficult after a traumatic birth. Second time around, I weighed my own child, happy for her to be checked if they called round, but I didn't take her to baby clinics (I tracked her weight myself).
I did have the three year (then) for the first, the second they asked me to contact them if I was concerned/wanted a visit, I didn't, and they were lovely about it.
It seems to have moved from an optional supportive service to a compulsory one. I don't believe it will help with child abuse whatsoever, all of the cases mentioned above, the parents and children were already involved extensively with social services/police and so on (50 odd contacts for one case) and it still didn't make any difference. Health visitors may be more useful for developmental checks, although mine couldn't spot a tongue tie and gave me out of date information about flattened heads.
If you talk to anyone who works in social services child protection or child psychology/psychiatry they will tell you there are so many already extremely serious cases overwhelming them and it is difficult to cope. My friend who is a child psychologist had cases of child self-harm, parental neglect and she used to have to beg ss to take the cases. I would rather the money was poured into these services than HV because with the best will in the world, a small visit once every two years is pointless as a child abuse prevention measure, Baby P's mother used to hide the child's bruises with chocolate and tidy up.
I don't think HV should interpret lack of interest in their visits as unwillingness to engage, the team I had involvement with were really lovely and saw themselves as offering services, not imposing them.