You sound similar to my mother.
Five children, countless pets.
'We don't do touching'. I have not one memory of being kissed, cuddled or shown affection. Quite a few of being hit and pushed away, though. And lots of being told that people who hold babies or kiss their children are weird.
She would talk about seeing people hugging or kissing their children, never mind friends, with bemused entertainment and criticism about how strange and twisted they were. I think these days, she'd use the phrase 'inappropriate' - with all the insinuations that entails.
You might wonder why she had five children. Well, she liked babies. Apparently because they go to sleep and you can put them in a cot. I think she saw the conception as being a necessary evil to achieve her little toy for a year/the point at which (as she said) they started answering back and wanting stuff off you - and the inconvenience of children being the price she had to pay to get her toy. For me, the animals were my source of touch. Probably why she got them, so she wasn't bothered by me wanting such weird things as a hug.
Every single one of her children was emotionally fucked up by her refusal to touch other than when hitting them. For me, it took until I was 16 to find somebody who wanted to touch me - had there been somebody who wanted to do that earlier, I would have willingly let them do anything to me in exchange for that physical sensation.
Your child is wanting your touch more precisely because she craves it. Because you deprive her of it. Because you are physically, and for her, emotionally, unavailable. And it sounds as though her father is trying to compensate for your coldness, but he can't. Because she wants it from you as well.
My mother was also convinced that she was a great mother. A natural at it, apparently.
So despite the PPs saying you are normal and it won't affect your children at all because hey, there's somebody else there to compensate for what you refuse to give them, I disagree. It is neglectful. It is abusive. And it will have longlasting effects upon your children.