I used to do home visits when I taught reception, so I can tell you a bit about what I did and thought, and I hope you find it reassuring.
The main aim, for me, was to get a sense of the child's personality, strengths etc. Some of this came from what parents told me, some from just seeing them play in their home environment, where hopefully they are fairly relaxed. I found it interesting to see what they liked to play with, and maybe get some idea of language skills.
All this comes with the big proviso that I knew children and parents were often a bit nervous and not totally comfortable with me. I tried to be reassuring, but even so...
I didn't care if the house was tidy, so long as there was a seat for me to sit on (bad back!) and so long as it wasn't truly squalid. I only went in one house that made me have safeguarding worries from the squalor alone, and it was really extreme. Mess, a bit of dirt, not a problem. In fact extreme tidiness could be worse?
I usually accepted a drink if it was offered, but not always if I had lots of visits, as I didn't like to ask to use people's loos (my issue, not to do with cleanliness).
I usually tried to engage the child in a game or story, watched them interact with siblings, toys, mum etc. Asked the carer if they had concerns or questions, things they wanted me to know in advance. Allergies, family issues, whatever they were concerned about. Sometimes we talked about pre school experiences / nursery / childminder and how that had gone.
I never went with a view to checking up or finding fault. The visits were brief (I had 30 to do) so it's very unlikely I'd see enough to worry me enough to make a safeguarding referral. Not impossible, but it never happened to me. Even in the case of the squalid house I mentioned earlier, it only involved me speaking to the safeguarding person in school, so she was aware. It's then her role to log info and refer if necessary.
Some families maybe I could see had problems or issues, but they usually became clearer once the child had started school.
The vast majority of families were welcoming, and I really enjoyed meeting careers and children. It got us off to a flying start in September, and hopefully forged relationships that were useful if problems / queries / worries arose later.
The visit I remember years later was to a family where the carer told me, in tears, that she could not read. I don't think she would have told me that in the school building, she was scared of schools. It was so useful to know that. We could help with issues such as letters home from school, helping her children with reading, checking that info was available to her in other ways. She certainly did a lot of phonics work with her child despite her own difficulties.
Anyway, I digress. I don't think you should worry. The people who worry are usually those who don't need to.