Either he had a father like this growing up (so he thinks it's normal and acceptable) or he's very aware that he's taking the piss.
My comments blow apply if it's the latter-
He'd never stop you from doing a hobby?? Ermmm, so when would you be able to do that hobby then? Let me guess - you do most of the childcare, which means you either don't have the time or you're too tired in the evenings. Lets say you took up a hobby that took you out of the house from 4pm to 11pm every weekend.....when would you two get to spend time with each other then, as a couple, as a family?
The reality of this happening is very small, and he knows that. Most hobbies and hobby social groups that would take you out of the house for this amount of time happen in the daytime. So.....would he be willing to sacrifice half of his weekends then so you can enjoy a 6 hr hike with a local walking group?
He hasn't verbalised it as such, but he's suggesting that you are unreasonable or controlling by suggesting you are trying to stop his hobbies. Useful tactic to keep you quiet. At the moment he has a nice wifey at home to look after the children whilst he gets the free time he wants on the weekends and check out of family life.
Where are your emotional needs in that then? Where are the needs of your children and you as a family? Most family days out with small children happen in the day time. Sure, you could try an shove in an hour or two at the local park together when he gets back, but is he even up for that when he's been out all day since the morning? What about longer trips to the countryside? Going to a museum or a different city for a day out?
What about being spontaneous and doing something when the weather's nice? Would he be happy to drop the golf so he can spend time with you doing something with the family?
If not, it's very clear where his emotional priorities are. He wants to have a break from all of you for hours every weekend, whilst you have too cope with the children alone.