I think the issue isn't so much one of neglect but that people don't appear to look at or even think about the bigger picture. Because in reality we do what we feel is right for our children now without contemplating how those decisions might affect them, and us, in the future.
I went to boarding school from the age of five. There was no choice; I had to go to a specialist school for the visually impaired in Exeter. So my first day at school was being driven on a five hour bus journey to a school notwhere remotely near my home, and having to share a room with three other children, none of whom I knew, but who had all started at around about the same time so at least we had that in common.
I was a weekly boarder, and although school life was good and we were in a nice nurturing environment, I never had the chance to go home after school, to my house, my toys, to do what I wanted to do. My sister didn't go to boarding school so I was essentially sent away while my family grew up away from me. It was necessary but that doesn't change the outcome.
Then when I was nine we moved abroad, and I was sent to a different school. This time it was different; it was a school where I didn't speak the language, so I spent the first six months or so unable to communicate with the majority of other children, and facing the reality that I either had to learn the language or I would be unlikely to make friends. I learned the language. 
Then when I was eleven my dad got a different job and my parents moved away so I went from being a weekly boarder to a term boarder. No twenty weeks holiday here; we had the same school holidays as all the other schools.
As I was now attending an Afrikaans speaking school I was essentially growing up in a different language and culture from my english speaking family, as I spoke nothing but Afrikaans when at school (still speak it fluently) and only spoke english to my parents when on the phone or at home.
As a result I am now nothing like my family. I have never grown up at home - I don't know what it's like to do so.
Whereas my sister has a close relationship with my parents mine is less so. Yes we talk and get on but I certainly don't have that closeness or the need to confide in them if I have problems, you can't take a child out of family life for the majority of its childhood and expect the relationship to stay in tact.
I certainly don't think it was neglect - circumstances meant there was no other option. And reality is that all the cool things happened on weekends so going home then means I would have missed out.
I have some excellent friends I made at boarding school and I am in contact with many of them now even, plus boarding school gave me an independence I suspect I may not have had otherwise.
But the relationship with my family has been permanently affected. And I don't know a single other person who attended boarding school that doesn't feel the same. And yes, most loved it, but sometimes by the time we can see the damage it's too late.