Hi, OP. Sorry if I say something strange, but as I'm not native english speaking, I'll try my best to make myself clear, ok?
I have a very similar experience. I broke up with DS' father when I was 2 moths pregnant. We were not married. He decided that if I was no longer to be with him, so he would not be there for DS.
DS is about to turn 6, and a year ago he asked me "mummy, don't I have a father?" I told him that he has, but doesn't know him, because we broke up before he was born, and that he lives far away (that's true, also).
I tried not to make this a taboo thing, but it's not very easy, since his father has absolutely no part in our life - so how to bring it to a conversation without being the subject? I mean, it's not like saying "look, it's raining, your father likes the rain!", or "you know, today I saw someone that looked like your father"... It just doesn't happen.
But the point is that children can be cruel sometimes, even without noticing. I've heard children asking him if he doesn't have a father, why did his father didn't want to be with him, and how could he be born with no father...
I think that the most important is to make him feel secure, knowing that you really love him, and that you will always be there to answer his questions.
And do it without making the father looks like a superheroe that wil be very glad if DS appears suddenly in his life someday! Because your DS WILL eventually want to know him (mine already said that one day when he grows up he is going to visit his dad so they can play football together...), and I can only imagine the father's reaction... 
It's not easy. Sometimes I burst in tears when I think of how to make it easier to DS.
But I try to keep in mind that I can make mistakes, and there is no "right" way to talk about it. I know I didn't manage to make DS as secure and light about the subject as I would like to. And sometimes I just wish we could be in a film, where there are always happy endings for these situations... DS would grow up to be a very hansome man, and would tell me "mom, I thank you for what you've done, you were the best mom, I never even miss having a father!"

Just keep straight. We all shout sometimes, we all get tired (specially being a lone parent).
And if you ever find out a way to make it easier... Please, post in here!! 