So you think there's only two types of time - leisure time and punishment? How odd. I don't think the fact that the OP enjoys baby groups automatically makes it leisure time - in fact I think the fact that she enjoys them means that it's vital for keeping her well, which surely is the most important thing?
To my mind there are two types of time:
Leisure time, which is about relaxation and choosing to do things that replenish my energy and interest - watching tv, seeing friends, reading.
Work time, when I'm doing things that get stuff done, such as paid work, house work, childcare, diy that sort of thing. I might enjoy the work, but it's still work.
If I had a third 'punishment' category I might say the gym! Or rocking a bawling baby at 3am.
I don't see time spent looking after children as leisure time, regardless as to whether I'm with friends or not, largely because it's hard to relax when children are around, especially small ones.
You see it differently. That doesn't really matter when it comes to the OP, because her DP is just a massive dick.