I've had the same conversation recently OP with my ds who is also 4. We'd really love another one but it hasn't happened and I'm not sure it will. A while ago I read a post on MN by an only child who said she'd wished she had know her parents couldn't have another when she was growing up rather than deciding to as she felt she would have accepted having no siblings more easily. This kind of struck a cord with me because we haven't chosen for him to be an only child. So when ds asked me if he would have a brother or sister I said in quite an upbeat matter of fact way that some mummies and daddies can have more than one baby and some have one and that if we did have another it would be nice but that we were so happy and lucky to have him. He seemed satisfied and I felt quite pleased I'd got through a much dreaded conversation in one piece.
And this seems to be a phase - the age of four is (as you say) when their friends acquire younger siblings, so it does prompt questions. We always took the same line as Bean and said in a fairly upbeat way that we would not be having any more babies.
It can be hard to explain to a little one (and even harder when you are still coming to terms with it) but we have always been quite matter of fact and just told DS that families are all different and come in different shapes and sizes and he accepts that. We just answer the question and then change the subject.
This is a good book for talking about different families.
DS occasionally mentions wanting a brother but usually in the same way that he wants a train/car/icecream!