Community and second home don't really go together in my head.
Community and holiday home? Yes.
As a member of a rural community, who lives on a tenant farm and owns a house which is a holiday let in my community which we sometimes stay in (even though it's four miles away but hey a change of scenery) no issue.
As an owner who lives in London, has a property here, employs local people, gardeners, cleaners, advertising etc, absolutely no problem. Owners and guests become part of the community.
If you mean a house to stand empty except for holiday use then yes I would probably momentarily think that's a shame, then move on with my life.
Would I judge, would the community judge face to face? Unlikely because we would assume you were tourists and be welcoming. You wouldn't necessarily become part of the community but only because nobody would know who you were