This is a tricky one, @KidsArt . On the one hand, you did undermine your husband's parenting decision, to teach your 3 year old that if he chooses to mess around and not get ready... the consequence is that he misses out on the swimming trip. However, as Tugboat said above, what would your husband have done, if you weren't working from home?
As unfair as this sounds, if I were your husband, I would have cancelled the trip altogether, and explained (gently) to your older son precisely why he and his little brother weren't going to the pool. That his behaviour, his refusal to get ready on time/at all means that everyone misses out. You're meant to be working, for goodness sake, not looking after a hysterically bereft small child!
I get how difficult it is to parent a child like your older son - my youngest was pretty much the same at that age. And yes; he has ASD. I've always used the "you make a choice, you abide by the consequences of that choice, be they good or bad..." method with my children - even at 2 years old, they understand. It's similar to the lesson most of us learned at a very young age that if the oven door is hot, we don't make the choice to touch it because the consequence is that we will burn our hand. So, for your older child, it would be that if he refuses to get ready to leave the house with Daddy and his little brother, for a regular (so he knew it was happening beforehand!), if he won't let Daddy (or Mummy) help him get dressed... then that's a choice he has made, and the consequence of that is that everyone misses out. Not just him; everyone.
Harsh? Maybe. But I'm a single parent and when my children were that age, I didn't have the luxury of leaving the one misbehaving at home whilst I took the other out for a treat. If my son misbehaved... my daughter missed out, too. And he soon learned to get ready on time so that he didn't have the weight of the consequences of his choice on his shoulders. He's 18 now (today, actually), and is one of the most punctual, responsible young men I've ever known. Your lad will be, too, when he's older and not in the throes of the misbehaving years, I'm sure.
I'd be having a chat with your husband, though, if I were you about not leaving one child behind the way he did, today. All that will teach your older son is that his Daddy is someone who will treat the younger child, whilst punishing him (however much a correction is needed, behaviour wise), therefore that's the one whom your husband prefers... and that could lead to a heck of a lot more acting out from your 3 year old over who knows how long. Also, I'd be reminding him that WFH is not something that you can do whilst trying to console a hysterical little boy who has no actual clue as to WHY his Daddy left him and took his younger sibling swimming! It's a both, or neither situation, regardless of whether you happen to be at home (trying to work).
Would you take one and leave the other, if your husband was WFH, I wonder...? Knowing that the one left behind would react - and understandably so! - the way your older son did? All your husband's choice will do is store up the consequences of damaging the relationship he has with your older child. Which I'm sure, deep down, your husband won't want in the slightest.
for you, because it is tough. But even though you did undermine your husband in his parenting decision... I think you were absolutely right to expect him not to choose between his children!