He's not being isolated. He joins in with all the singing (😫), dancing and playing of games. How exactly is he being isolated?
Because they are gossiping to group themselves against him. Just because you're allowed to participate doesn't mean you don't know that the group is talking about you (as Seann said) and pulling you down.
They're all so aware of how he is/was perceived on the outside and are hyper-aware how it might reflect on them if they get too pally with him. Some might say that makes them cowards. Some might call it self-preservation but whatever it is they have their guards up.
No one is suggesting they become best friends, though. They just need to accept his presence, which as far as I can see isn't an overwhelming personality. Surely they do research before they come on and their agents tell them that if it's seen that peoole are being excuded, it backfires on the excluders. Hence Gary Busey winning CBB (who must have been disgusting to live with!) Just being polite to people and not gossiping never lost any celebrity votes ever.