I think that children WILL ask questions about each others lives, from the age of 4 or so, and that is absolutely fine.
Obvs, if a child is being actually rude, they need to be be pulled up on it, but simple curiosity is fine.
It's good to let children know that all families are different, just like it's good to let them know that people come in different shapes and sizes etc .
However, in your case OP, I do think that it's you who are feeling a little sensitive on the subject of your ds's dad, and are understandably worrying about him feeling different.
I agree with Kewcumber in that it is your responsibility to talk it through with him, and find out if he has any questions. Also, to make sure he knows that he can ask you anything at any time.
What you don't want, is to build his fatherless state into a big thing, something he senses he should be ashamed of.
I am very casual with my ds about the fact that his dad doesn't live with us, and doesn't see him much.
It's just how it is. When he was small, if he encountered a dad at a friends house, he always assumed they were visiting! When he started school, and realised most kids live with their dads, he did feel a bit strange about it, but again, I am very matter of fact, and don't make the when we discuss other family set ups, and our own.
I used to feel a bit uncomfortable meeting other school mum's etc , being the ONLY singleton in ds's class, but the better I have got to know them, the more I can see that a "normal" set up can be quite strange too!
Sorry, rambling, but do you see what I am saying?
Yes, it's nice when parents teach tolerance, but you have to be comfortable in your own skin, and your own little family. Then ds will feel comfortable too, and it won't matter what anyone else thinks.