Perhaps you should try keeping a diary for a month.
Place it into something similar to skybluepearl's categories.
One colour for when you're both working or commuting to work. Another colour for family time. Another color for family chores (e.g. housework, babysitting, taking them to appointments).
The final colour should be his time, just for him. And then mark up your own leisure time, just for you.
If he's being such a selfish "weekend dad" (i.e. the good parts, none of the crap parts) a short sharp shock might suffice.
Then you need to ask him if he agrees you should roughly have the same leisure time. If he agrees, excellent - you just need to come up with ways using the week chart where you re-jig stuff. Be that you taking your Saturday afternoon of leisure time whilst the next week it's his, in turns.
But if he basically sees the inequality on front of his own eyes, and doesn't agree that you should sit down there and then to re-jig the normal week's schedule, you'll know you've got a DH who not only realises there's little leisure time for you, but who is also happy to let things continue that way. And you've got bigger things to worry about than the minute details (e.g. that he's off to golf again).