I realise it's a niche market, but surely it could work. There could be lots of vigorous mucking out to burn calories, plus grooming etc. Mixed in with several hours' riding a day, it could be very effective.
I know some stables do "riding for fitness" but I'm thinking about a week/weekend away.