From what you’ve said about your situation, I can see why an HV might think you could use more support than most people. But you clearly aren’t finding her supportive and she’s told you she doesn’t have any concerns.
So I think you should call her up and either tell her you no longer want an HV or ask for another. If she tries to put you off tell her you’ve found her visits time consuming without being at all useful and that her behaviour coming to the house after you canceled was rude, so unless she can suggest a different HV with a different approach that’s actually supportive and helpful you’ll end your engagement with the HV service and you’ll just go to the GP if you have any concerns about your DC.
Engagement with the HV service isn’t mandatory, whatever she’s said to you. She could pass on concerns to SS if she has any, but she says she doesn’t. If she threatens you with SS if you end engagement, you’re probably better off. At least that way you’ll find out what the real concerns are and have a chance to demonstrate they are unfounded. I suspect, though, that SS will be too busy to do anything more than a cursory visit if they get a referral that boils down to - “I keep going round but client doesn’t want to see me”.
One caveat I would add, though, is that HVs can be a route to support if you’re struggling. And you sound like you have little support. So if you are struggling at all, or if things are okay at the moment but they could go south at a moments notice, it might be better to try and maintain a relationship with the HV. But perhaps start making it more on your terms and tell her “No.” if she wants to come more often than you want to see her or when it’s not convenient. Ask her to leave if she’s over stayed her welcome. If she calls round when you haven’t got an appointment, call her out. Tell her to stop trying to catch you out with surprise visits, that you deserve more respect than that. Just remember - You have the power here. She has no authority at all.