The OP asked about 'sneaking off' to the gym, so clearly is not referring to environments that many have where it's fine to work flexibly and take time away from work during the day.
I manage a team of 8 and we have a hybrid working arrangement, 2 days WFH, 3 on-site. Pre-pandemic we had a flexi time system where you could start between 830 & 10, finish between 4 & 630, but business need had to be prioritised.
Once we moved to the current arrangement there was initially no flexi time & an attempt to have standardised working hours, which just didn't work & it was silly to try.
I'm fine with colleagues taking time during the day, but a gym trip wouldn't be ok, unless in lunch time. My team need to be contactable throughout the working day to ensure collaboration & team working - so if someone was unavailable in core hours for more than 15 minutes (and not in a meeting or with some indication on their calendar) that would be an issue, and is one I've had to deal with as their manager.
I do have more flexibility as a manager as I will work late / weekends as needed so can fit in a personal task on occasion. I'd rarely do it. Likewise I need to be available to my colleagues as much as possible.