I suppose the HV's thoughts will be v much with your little boys, and what is happening to them.
The SENCO, and your lad's teacher, will be in a much better position to make sure he has the right support if they know truthfully what has been happening to him. I think the HV is right to press for that meeting.
Also, the more you talk to other people about it, the more you will realise you CAN talk about you, your needs and your DS's without H being part of the conversation. My guess is that you may start to find it very liberating. Look how you have progressed this week - two difficult meetings! it can only get easier, and you will be the stronger for it.
It was blunt of the HV to let you know what her gut feeling is, but i bet she has heard many many stories like yours, and has seen what has happened.
To be honest with you, my own feeling is that you need to leave this man. IF he takes responsibility and gets help, it can be a 'separation', and if he is genuine in his realisation of the damage he is causing your little boy, he will actually understand that it may be better for you and the boys to be away from him until he has made some progress.
If he makes no effort at all - and doesn't make a move to get help, I think you should move, swiftly.
That's merely my opinion from a distance (but you are very effective in the way you describe things) and obvioulsy you will do what is right and best for you and the boys. Didn't want to hide my own 'take' on things, any more than I think you should be 'persuaded' into anything by us MN-ers.