Yes, sadly it happens and not much you can do about it.
I've working in five different small accountancy practices, doing the same kind of work. The different between firms for no obvious reason is staggering. In a couple, I fitted in straight away - most of the other staff were just like me in terms of work ethic, personality, politics, interests, etc. In the other three, it was the opposite, most of the other staff were completely opposite, different personalities, different interests, different politics, etc.
It's just luck of the draw really. The first time it happened, I stayed, persevered and tried to fit in, but it just didn't work. The last couple of times, I realised virtually the first day and made pretty quick exit plans to get out as soon as I could secure an alternative. Sometimes, personalities just don't fit, the "newbie" who doesn't fit, will eventually move on, leaving the others to continue working happily together.