I assumed it was a 'local' thing and a one off, until I lived in Gwynedd for four and half years... I was surprised to be greeted by 'GO HOME ENGLISH BASTARDS' spray painted across the road in front of the university in foot-high letters when I first arrived, and then the language switch happened several times during the years I lived there.
That was one sad individual/small group, though - you get those everywhere. If they were actually locals, they would know that Bangor University has students from all across the world (and Scotland/NI) - not just England. It wasn't a project that the local council put to a public vote and then organised once it was carried unanimously!
I learned a (shamefully) small amount of Welsh when I was there, but always relied on being able to communicate everywhere in English. I never had the slightest issue. The only ill-feeling I observed was when certain boorish people treated the Welsh language as a silly hobby that the locals persisted with, just to be difficult.
As a PP said it's quite easy to notice a change in tone, cadence and rhythm when the spoken language changes, it completely different to the mixture of Welsh and English the passengers on the bus used, for example!
I did indeed notice that first-language Welsh speakers had a different cadence to their voices, but many of them had the same cadence when they spoke in English. They spoke using perfect English grammar and vocabulary, but rhythmically, it still sounded Welsh, IYKWIM. I have a very good friend who comes from South Wales who speaks even less Welsh than I do, but he still has the same cadence. A lot of it is purely down to Welsh accents rather than the actual language they're speaking. Listen to British people speaking French or German - the vast majority who have learned diligently might speak the language perfectly, but they still don't 'sound' like a French or German person, beyond the actual words. Many don't even realise that there's a difference and presumably just think it's individual accents.
Also, aside from the fact that your average Welsh person and average English person really don't look noticeably different from each other, as you would guess, the Welsh language is most prevalent in parts of the country that are well inside Wales - you don't get many FL Welsh speakers in the borderlands. I don't deny at all that there are unpleasant, hostile, insular people everywhere, but how could such a person in Gwynedd or Ceredigion be so certain that their 'foreigner' wasn't from another Welsh-speaking village, just by looking at them, before presumably starting to make rude, insulting comments about them?
You get nasty people everywhere, but I don't think some English people quite grasp what communication is like for a lot of FL Welsh speakers. Just because they all speak English perfectly, it doesn't mean that it makes no difference to them which one they speak. As you say, they will almost always graciously speak in English to accommodate those who don't speak Welsh, but it doesn't mean that they don't still think in Welsh or that, however perfect their English, they aren't more comfortable in their own language.
I've observed certain English people look gone-out at a FL Welsh speaker who is speaking to them in English when they take a second to think of the English word for something or momentarily stumble, as if they must be completely stupid - oblivious to the fact, if the other person were speaking their 'own' language, they themselves wouldn't have the remotest clue.