I think, if I don't give many downsides or pitfalls (and I honestly did list all those I could possibly think of), it's because generally I know the girls are really happy and I wouldn't change what we're doing for the world. It's not through any kind of zeal to convert anybody, and it's certainly not through any kind of hiding the truth - the absolute truth is that HE is the most wonderful thing that ever happened to us.
I think that the only reason HEers will sometimes defend themselves against downsides non-HEers give is that, like MrsWobble says, these are often based on assumptions or myths and not based at all on the reality of HE.
Having had children who have been in school, one until the start of Year 4 and one right up until the start of Year 8 (so all through primary school and some of secondary), I honestly don't think there's anything about school to miss. That's not being anti-school - my eldest had a lovely experience at infant school in particular - it's just thay I don't think they got anything in school in social terms that they aren't getting the same or better with their home ed friends now. The trips we go on are very similar to school trips, we meet for group classes in all kinds of stuff, and the children in the groups are definitely not all children that my DDs would choose to be best buddies with, so that same negotiation and learning to get along with others still happens. The only thing I think they miss out on is bullying, the kind that is incessant and drains every bit of confidence from you. Ny eldest had terrible, terrible bullying in Year 5. We kept her in school and worked through it as best we could, but it was the wrong decision. She isn't a better or stronger person for it, despite those who would say such a thing is "good for you". My youngest, who came out much earlier is streets ahead in terms of confidence and getting along with the outside world - she's just less damaged by school.
I do know what people mean - virtually everyone has gone to school and it can be hard to have had a completely different childhood to everyone else - you don't have those social references that most of the country have. But if my previously schooled girls are ever asked if there's anything about not going they feel they are missing out on, they are always emphatic in their "No" - and they've been home ed for long enough now for the novelty to have worn off.
So, downsides? There are some genuine downsides sometimes, and I listed what I could think of. But if those seem small or insignificant it's not an attempt to convert or mislead - it's the truth of what HE is like for us as a family.