Hmmm.
He agreed to the exchange. Fine. I would assume an exchange to be a couple of minutes, go in, pick up another item, take to till and done. He probably did too.
He gave 'helpful' opinions on a couple of outfits, which covers the 'taking an interest' side of things.
OP chose some clothes....
Now call me weird, but isn't that the point when an EXCHANGE takes place and you leave?
Exchange = go on, pick replacement item, go to till, leave.
If that had happened OP would have got her ice cream. But no.
At the point where the till was next on the list, after 20 minutes spent looking for items to exchange, that are now in her hands OP utters the line:
'I'm just going to take one last look over that side, then I'm done.'
The 'last' implying there's already been a first second third fourth look over that side.
So the DH offers to take everyone out for a nice family evening after a hard week all round. OP wants to exchange some clothes, DH agrees. OP then spends 20 minutes looking for items to exchange, during which, despite being 'antsy' DH takes an interest, discusses the outfits etc. OP chooses the items for exchange so has done what she'd said she wanted to do, and took a while doing it, then declares that, even though she's got what she came for and she knows that DH is antsy she's going for a last look at the other side.
I love shopping, but if I had gone for dinner with someone who said that they just wanted to exchange something, then spent 20 minutes bimbling around a shop, chose the exchange items, but instead of doing the exchange and leaving, carried on shopping even though they'd already looked at that area I would have been annoyed.
You took the piss OP. You knew he was getting antsy, but he still took an interest. You had chosen your exchange items, knowing he was getting antsy you should have gone to the till. Instead you decided to have another bimble about, looking at stuff again.
Yup, you took the piss, and in his shoes I would be fuming too.