I do a hobby with others every Saturday. Out of the group, maybe a third might go for a brew/drink afterwards. We've spent a full day together. I don't always go for a drink as sometimes I've something on/ have things to do at home/ more often than not I just want to get home as I've spent all day with these folk, nice and all as they are, and I've nothing more to say to them.
So, others go always, some never go and I go sometimes. Someone who'd never miss the drink or coffee afterwards has said some of us are antisocial. Are we?
YABU - I'm antisocial and should go on every occasion.
YANBU - I've spent all day with these people already, enjoyed their company, but occasionally want to leave straight after and that's okay.