But he’s saying that I didn’t want to go in the first place and wasn’t supportive - which causes him to question my loyalty to him.
So unsupportive that you came up with a solution that would have worked for the entire family - not just him - and even went so far as to gain agreement to work remotely for periods of time to make that solution a reality?
By his logic, the only way he would have considered you "supportive" and "loyal" was if you'd disrupted critical years of your child's education, jacked in your own career, and uprooted yourself from all your support networks.
That you even considered doing any of those things for him is more than most would do. Frankly, very few would want to do that, and you had every right not to want to do something that would have had a detrimental impact on everyone in the family except your husband!
So who's the unsupportive one here?
Sorry, but he's just mis-using "loyalty" as a synonym for "obedience" to manipulate and blame you for something that wasn't your fault. He's upset because you didn't obey him, not because you didn't support him. And that is not okay.