Does the gym have a creche to pop the kids into? Do you have a car so you could pop in with the kids? Do you not live near any family members who could take the kids for an hour? Could you afford Sitters? Do you have a friend who could pop round to where he says he is and 'bump' into him. Are you unable to leave the house if he's not there (ie no car or public transport) if not, then you can go places he claims to be. "Just took the kids for a walk/drive, thought we'd pop in and say hi"
So many options.
I find that if someone is trying to hide something like that, they will betray themselves in their response to the idea of you turning up! So even if you can't get a babysitter, you could phone him and tell him that you are going to ask your brother/sister/mother/friend to babysit and you can join him straight away, and where is he so you can get a taxi. If he's pleased, then great (and you can phone back and tell him soandso wasn't answering the phone). If he's terrified, you have your answer.
Of course, that's all sneaky and not the best way to conduct a relationship, far better to lay your cards on the table! Sadly, many men lie through their teeth even when confronted.
But these are so many assumptions. Maybe it's a midlife crisis, or problems at work, or someone's making jokes about him getty pudgy or a million other things. Perhaps he's depressed, or stressed?