Really great advice LittleDucks.
I didn't enjoy having HVs in my house, but while I have a very open door house where DP's friends/ my friends are welcome and have loads of visitors, was a bit creeped out by H/V visiting. Totally different with midwife who had monitored me and both DCs, and since she lived down the road, would do ante natal home visits.
but the H/V felt like a stranger, and she was for all intents and purposes. I asked the midwife not to discharge me for as long as possible, and then the H/V visited a few times for DD, and only once for DS, when I said that I would prefer going to clinic.
Don't think you should refuse, but at the same time, let them see you are OK, the baby is OK and you are coping (if that's the case).
But at the same time, consider the experience of my cousins in the USA. She had twins by c section and was discharged after 2 days. The babies needed to be checked every day, so they had to go from their house to the doctor, which was a 30 minute drive. Get checked over, weighed, and then they would go home again. They were so jealous when I told them that we were visited at home and didn't have to go out for the early monitoring.
Your call, but the H/V home visits can save a lot of hassle.