well fwiw I think this is all down to gut instinct. If you feel it's wrong, and you don't generally have an issue with this type of thing normally then it probably is wrong. But I take bubblegum's point, it is tragic that men are often cast in a suspicious light if they are chummy with kids.
My bro has a friend who is the life and soul, always been with the same woman, got married to her a couple of yrs ago and they have a longed-for 6 yr old dd. He totally "gets" kids, he is a kid, very funny, natural comedian type, very self mocking and always putting on a turn for adults and kids alike.
I know he sounds irritating, but he's not, amazingly. And children just love him. They flock to him and he manhandles the big ones, cuddles the little ones and coos over the babies. I've seen it several times and it's not remotely creepy, and I think I have a good radar for that sort of thing. My chidren love him (but really, they don't know him well), but if I thought for a second they didn't, or there was coercion on his part or something else going on-that would be that.
I suppose the point is-it can happen, but I'm guessing this bloke isn't like that? You need to go with your instincts.