It's unacceptable to use racist terms in any scenario, I don't use racist terms at all.
I wouldn't use them in relation to gypsies / travellers.
However, unfortunately op, a small number of travellers (Irish in my experience) do give travellers (of all origins!) a bad name, because of some of their behaviour. This of course, does not justify anyone hurling racial abuse at them or others, but I can understand people's (non travellers) frustrations because some of the behaviour I have witnessed is unacceptable.
Having said that the behaviour of a lot of non travellers is unacceptable too.
It's unfair to tar all travellers with the Same brush obviously, but I can (unfortunately!) see why it does happen.
For instance, in the village I used to live in a train from a garden centre got stolen by travellers. It was there for 20 years running children about the garden centre before it was stolen.
If there was not so much bad behaviour I think there would be less hatred and racial abuse to be honest, though like I say there is absolutely no excuse for racial abuse. It's okay to be angry and frustrated at bad or anti social behaviour, but it makes you just as bad as the person you're angry at the minute you start hurling the word 'pikeys' about :(