As my own children were older when I started minding, I had experience of all age groups, so certainly haven't found any ages being a problem. I guess it would be a bit more difficult to have a much older schoolchild if you don't know anything about that age, but it's not rocket science and you'd soon learn. Lots of other childminders mind older children and only have toddlers of their own, so you can make it work.
I love looking after babies, but I also love them learning to walk, becoming toddlers, and then seeing them off to school. Some childminders don't have babies, others won't have over fives.
As HappyMum says, I think you'd soon find that staying indoors all day long with a child would soon tip you over the edge! Going to a toddler group or soft play session is good for your sanity, not just fun for the children.
You don't have to drive to go out and about, I know minders who use public transport! Do you have a school which is walking distance? If you had children before or after school, you would have to go to the school to drop off and pick up.
Also agreeing with HappyMum, you don't have to take on anyone you don't want to.