Some Monica Edwards Romney Marsh/Punchbowl books are very collectable and prices reflect that. You can probably get Wish for a Pony cheaply, though.
KM Peyton has written some horsey books for younger readers, and then she can move on to Flambards, Fly-By-Night and so on when she's older - some are definitely YA rather than children's books (and those are the ones I know better.)
Others you can probably find cheaply second hand include Monica Dickens - Follyfoot series, Patricia Leitch' Jinny books and Ruby Ferguson's Jill books.
I think I started reading Elyne Mitchell's Silver Brumby books around 8 or 9, and Mary Elwyn Patchett also wrote the Brumby series. (My heart is always with Thowra, though, and ensured the Snowy Mountains were on my itinerary when I went to Australia.)
You can probably pick up Pullein-Thompson sisters' books cheaply. Also Pat Smythe, who wrote the Three Jays.
Walter Farley, the Black Stallion series.
Marguerite Henry, Misty of Chincoteague series.
Mary O'Hara, My Friend Flicka series.
These are all classics - I don't know much past the 1980s, but there are more recent series. You can get some anthologies of short stories, too. If you Google, you can find lists of series and standalone, and fan sites. (And collectors' sites, if you want to get really obsessed and spend loads...)