Very much depends on how said and how it fit in to other things.
To me the word attractive doesn't have any sexual connotations.
And it sounds more bland than beautiful, pretty, stunning , handsome etc.
I am sure the vast majority of adult men have the ability to recognise that a young girl or young boy is attractive/pretty etc without meaning anything more than that.
I bet a woman wouldn't be given the same concerns if she commented on the young girl (or young boy) to say he was attractive/handsome.