I used to say this a lot to my friends if they were mumbling or using slang I didn’t understand. (I would never use it to someone who’s first language wasn’t English!). My parents also used to say it to me when I was saying like all the time or slang terms.
I feel like actually maybe I could be being offensive accidentally (I have ASD so it’s hard for me to gauge) but I wouldn’t want to offend someone by mistake. I’m aware that whilst I wouldn’t say it to a non native speaker I could be overheard and someone might take it personally 
This isn’t meant to be goady or racist at all so please delete if anyone feels it is.
Should I stop using this phrase?
YABU - no keep using it
YANBU - stop using it