There is a lot going on here OP.
Firstly, in your OP you said "I figured as adults they would figure out the truth and that would be that."
What did you mean by that - were you expecting them to totally disown their dad and be 100% by you? That's the impression I'm getting, but I feel like this is an unreasonable expectation. Why do they have to choose between you? They can have relationships with both you and him, and his family. Choosing to see them and spend time with them and yes, even live with them, does not mean that they are rejecting you. I understand that you're feeling it as a rejection, but I'm sure they are not intending it that way. Perhaps, even, choosing to live there is feeling like the best option in terms of encompassing all of those relationships if you have been hostile to them contacting their dad/dad's family while living with you, but he hasn't been openly hostile about them having contact with you while living there.
Secondly they're not really adults yet - the fifteen year old definitely not, and twenty is on the cusp of emotional maturity as well. Some twenty year olds are very much there and some have a way to go yet.
Plus, if you've done things right, then they shouldn't feel that they have to choose, in which case it might seem perfectly natural to them to make decisions based on whose house is more comfortable or which has more people etc. That's not a reflection on you, at all. In fact it shows that they are not game playing, really, which is a good thing. Again it feels like a rejection, but that doesn't mean that it was intended as one.
I think that the "loyalty" thing is a bit of a red herring. Being loyal to you means loving and supporting you and keeping your secrets. It doesn't mean that they must eschew everyone who has ever hurt you. Try to remember that their relationship with their father, grandparents and extended family is totally separate from YOUR relationship with your ex (and family), and again not a bearing on their relationship with or loyalty towards you.
You say that your son was guilted into living with his dad. So that might be playing a part as well. Again, fifteen is not emotionally adult. It's difficult even for an emotionally healthy adult to resist manipulation from somebody they care about - it's extremely frustrating and upsetting to witness this happening to your son, especially in this in between age where he's technically old enough to make a decision, but not yet wise to his dad's games, and perhaps not confident enough to challenge him. But it's not his fault, and it's unfair to him to blame him for it and feel angry with him. Feel angry with your ex, if you must, but don't blame your son who likely feels very torn about the whole situation.
In your last post, you mentioned that your son acted "like a jerk" at five years old when you divorced. It seems as though this is painful for you because it feels like history is repeating itself. When really, I think that you have misconstrued your son's motives. At five, you have a very confused, hurting and upset kid who is hitting out at whoever is closest and whoever is to blame. That's really hard to deal with, I know that. I also had a troubled five year old (he is older now). But do know that although his behaviour seems personal (he may even say/have said it was personal), it's just the way that an emotionally immature (as all five year olds are) person deals with hurt and strong emotion. Honestly, it was nothing to do with the performances. He might have latched onto that as a reason, but that would not have been the actual reason he was hitting out.
I can totally understand why you feel hurt and rejected right now. I think it would be a good idea to hold back a little, if you need to do that, but I wouldn't disown them completely. As hard as it is, you have to be the adult and once again take on board that they are still young and you have to give them the benefit of the doubt always. That's just how it is, your job, and you might have to do that for many more years yet. I can tell you for sure that if you reject them openly right now, it will be extremely difficult to get them back in the future. It's fine to tell them if you find something they do hurtful, but do understand that their relationship with their dad and his family is actually their right, and none of your business, whether you find it hurtful or not. That's one hurt that you just need to accept and/or try to deal with yourself without involving them.