It wasn't a problem for you, but it might be for someone else. Given the amount of vocal public antisemitism from some doctors, it might be a problem for some Jewish patients.
NHS employees shouldn't wear religious or political symbols. Many female doctors wear hijab now, but until very recently it wasn't common - most Muslim women in this country didn't wear it. I had many Muslim friends when I was at school, and none of them wore it.