As far as I can see, he's not doing anything wrong here. He's not leading you on, he's not promising you that this relationship is going anywhere further etc.
He's telling you "This is when I'm available, if you want to fit in with that, great If not, then I'm not bothered"
You are the one who wants more from this relationship than that. If what you've currently got isn't enough for you, then it's on you to make the decision to end the relationship.
And this last bit isn't aimed at you OP, but the people turning their noses at other peoples hobbies - What the fuck is wrong with Skateboarding? It's good exercise, gets you out in the freshair, takes skill, can be fun (I presume, I've never managed to stay on one for more than 10 seconds without falling off).
Given how huge it was in the 80s, I'd imagine there's a fair few skateboarders in their 50s around. You wouldn't go "Eugh, why's she still horseriding, doesn't she know she's 50" or "Ugh, she's been ice skating since her teens, she's so childish"
Just because a hobby is more stereotypically male, it doesn't make it more childish.