I think most people have doubts - they are just outweighed by the negatives they see in continuing with school, so you learn to deal with whatever you think the issues are likely to be. It depends a lot on your child, your situation, why school doesn't suit, etc, etc, etc. But, in general, the things people might worry about can be got round with a bit of imagination.
eg sport - ok your child won't be in a class doing PE. But they can still join sports clubs out of school, have a kick-about in the park, join a gym (older ones - here they can use the cardio machines from 13 I think), cycle or go out for family walks
working in teams - again, sports clubs do plenty of that, so do cubs/brownies etc
You don't need to cover specific subjects - if you do things that cover numeracy, literacy, finding out about how the world works, and something creative, then it doesn't much matter if you don't cover eg "history" as a specific thing. And if your child develops an interest in something later, you can find out about it together.
The main cons are really about having to keep explaining/justifying your choice to people who don't know anything about it and refuse to learn or to believe that you might actually be able to tell what works for your own child!