Hey OP, a social worker here. First point to make is different social workers do things differently so there's no one rule about whether they'd speak to you over the phone, visit or where they would ask to see in the house. Good practice would be to visit you to talk face to face, and of course to see the child (this is important because the child is the reason for the referral and has to be the focus)
From what you've said, they're not concerned about you smacking your child in a way that is illegal, ie causing injury, because as you rightly pointed out if they had he wouldnt' have been allowed to go home with you without a medic checking him over.
If I'd been told what you're child said i'd be wondering about a couple of things - is he wetting at home and is he getting into trouble for it? Are you struggling at home with a new baby and an older child and should any help be offered? Are you smacking and if so is it because you don't have an alternative way of dealing with it? Is there something else going on and your child is trying to get attention in some way?
Please note none of those are accusations, but they are plausible issues from the comments made that i'd want to consider and either rule out or try and help.
One thing to bear in mind is we know children sometimes make things up. We don't focus on 'proving' - we look at what's plausible. But its not as simple as 'a smack = bad, no smack = fine'. For example, children who are anxious about something or upset about something themselves might say that an adult shouted at them, when nothing was said. It can be a way for a child to express the anxiety or upset they felt by transferring it to someone else. With regular wetting, lots of children find it really embarrassing and can get into a vicious cycle where they try and ignore the issue and it makes it worse.
The standard that social workers have to look at is 'good enough' care. The threshold for that (directed by the family court) would be seen as remarkably low to most parents. We aren't checking for show homes or perfect parents or one way of bringing up children, we're checking that people are getting the basics right to keep the alive. Beyond that we'll offer support if we've got services that can help, and often that's not about there being problems with parenting but ways to make a tough job harder (eg support around routines and boundaries, because the difference between an Ok routine and a good routine when you've got multiple under 5s for example, can make a big difference to how much sleep a parent gets!)