I don't like it. It's fishy.
He doesn't want to look after the youngest, but doesn't like it if you take all the children with you, doesn't want to come to the stables himself either.
Wants to do his own hobby, but not while you're at the stables with all the children, what's that all about? Blames his lack of hobby time on the pony.
It's bullshit.
Probably you're right about the jealousy part, but not the reason. Jealous of the pony, literally, everyone loves the pony, everyone fusses the pony, the grumpy arse sat at home skulking in the corner feels like he doesn't get so much attention any more. Its common, especially where people's DP is a bit of an arse. The decent ones find a way round it, get their own hobbies and are glad of the free time, or join in in some way, maybe accompanying rides on bicycle or just pitching in to get the work done quicker so everyone can go home to do something else.
I can't stand the ones who say they want to come then stand around bitching about the cold (yes that's what happens when you do nothing, work keeps you warm) or constantly saying hurry up when they're the reason it's taking so long (work is obviously quicker with two doing it).
Sometimes it's not really about the pony, it's about you having a life that doesn't revolve around them and their ego can't deal with not being centre of your thoughts at all times. Stick with the pony. Try to pin your DP down on exactly what he wants using tons of questions and hypothetical scenarios to clarify, if he tries to be vague, so you can compromise where possible. But you'll maybe find he doesn't want anything specific other than you all at home with him and him having freedom to do his hobby whenever he wants IE totally unreasonable, there's a lot of it around!
If you want time away from the stables you could try full livery, part livery, getting someone to turn out or fetch in for you at one end of the day so you only visit once daily or getting a sharer for a few days per week. All these options have their downsides, eg expense incurred by paying for services, accidental expenses/ill or lame pony caused by people inevitably cutting corners from time to time and the possible impact of that on the pony's health, bad manners creeping in if others handling are not keeping on top of any cheeky pony behaviour, damaged equipment through carelessness etc.