You can decline, but in this situation, I would not (yet). If your HV has told you she is concerned (I know you think she's being ridiculous, I had similar with an overbearing HV myself), it is best for you to start attending the weigh in clinic weekly. That way, you don't have to have her in your home, and they can still check baby to show there is no reason for their 'concerns'.
When you go, just casually mention that you and your husband have both been ill, which is why you didn't have her around. Then say you're happy to come to the weigh in clinic. Then, once you've been for a few weeks, say to them, as there's no weight concerns, is it ok if I come monthly? They may ask for fortnightly, but you can either agree or show them the bit in the red book which states that babies without weight concerns shouldn't be weighed regularly (think it says more than monthly but check).
Then, it'll feel less pressured.
You can always say after a number of months have passed, "oh wee Jimmy's doing so well with his weight gain, I don't need to keep bringing him here every month, I'll see you at the annual development check appointment, thanks for everything".
Use your judgement to see how they are as to whether you could opt out altogether or not. The problem is, although you can opt out, as soon as they've said they "are concerned", if you opt out, they will call social services. I know you may think that's ridiculous, but some of them really do get concerned over precious little.
My last HV was very overbearing and insisted on coming into my home despite me having highly contagious gastroenteritis. She then wrote in the HV notes that "mother did look unwell, so perhaps she is being truthful". Cheeky bitch. Not long after that I made a complaint about her (there were a lot of reasons, she really was awful), she got very spiteful and started sending letters out to the GP about her "serious concerns", this was only AFTER I'd made a complaint about her and asked for a different HV. So I always think you need to be very careful with these people.