OP, I fully understand & get how you feel.
You are trying to see things from both sides right now, it hasn't happened before so it's easy to make excuses.
Coming on here and asking for advice is probably the wrong decision, too many people on here who judge without knowing anything, humans for you!
Of course, no one knows how hard he hit your child, exactly where & other key info we'd need to make an informed decision.
Obviously by you saying that your son became drowsy after, it could have been from being hit depending how hard however it could have been from crying.....
I'd say take him to hospital......what a lot of people do not know is that no matter what a child is taken to a&e for, whether it's suspicious or not, social services are informed each and every time.
As you are the parent who's left your home and now will be taking him to hospital, there is absolutely no reason SS would remove him from your care.