I always had ponies as a child. Up to about 15yrs i think when i swapped ponies for boys.
I always had mine out all year, in a NZ rug, but with an open fronted, timber field shelter. I used to have to cycle outside of our village in the dark before school in the winter with some warm mash and a hay net for him, every morning. But i didn't care at all!
Never had a tack room, the tack and hay bales were kept in a shed, never had a horse box either as i only used to ride locally and attend local gymkhanas that i rode to.
The main expenses for us was field rent, 6 weekly shoeing and 6 week'y worming, plus winter food. Yearly vaccinations.... can't remember much else.
You CAN keep ponies reletively inexpensively, if you have to, or you can spend shit-loads of money, its up to you and what you want and can afford.
I do think its great for children, although when i was a kid, we wore ill-fitting hats, tshirt and jodphurs and just bombed off in large groups of kids across fields and woods, making jumps out of branches etc with zero supervision and it was great fun!