I've found that those who think people can only have a good time if they drink alcohol are actually pretty dull themselves
so ignore the mean comments. It was nasty of your colleague to say that.
But it does sound like you need to start doing things for yourself too. And that absolutely does not mean doing something that doesn't appeal (eg going out drinking) - try and think of something that YOU really want to do, eg:
Join a choir
Evening class
OU course (I think they still do 'openings' courses which are not at all pressured)
Book group
Gym/fitness classes/walking club
This is a subject dear to me as I've learnt since becoming unwell that it is vital to my health (mental and physical) to do things for myself. I never had this before, I always felt like a non-person (in fact it was a big factor in me deciding to TTC when I was very young - I didn't want to care for myself, only someone else - not that I regret it now of course, my DCs are fabulous :o but when we have another DC it will be very, very different). I now have a few different things I try and fit in - piano, reading, maths. Have done art classes in the past and would love to join a choir. When DCs return to school I'm going to start learning to read/write Japanese and reading about a particular subject that fascinates me (history of animation - I'm a cartoon/anime geek :o and these things develop naturally, so if you don't have any particular interests now, it doesn't mean you never will!).
Hang in there. It's bloody hard when DCs are little and it can feel impossible - and indeed futile - to try and carve out any 'you time' but it is really important. It gets easier once they are at school.
Please don't worry about what your immature colleague said 