Trust me, trust me, trust me, trust me, TRUST ME...
Dump him now! Nothing better than you going to your parents and telling them that he wasn't good enough for you and that any man who would give you an ultimatum like that is a prick and not worthy of an invitation to your parents for Christmas.
Never, ever, EVER have a child with a "man" this bloody shit... Not only will you hate him and hate your own life, you will never ever EVER forgive yourself for inflicting a poor innocent child with a wanker like that for a father.
My ds is almost 10. After almost 5 years his dad's been back in the uk and managed to disappoint him by not being remotely bothered about taking him to do anything fun, has laid about doing bugger all and been utterly hideous to me.
Have faith in yourself, don't settle for this shit man. If you are concerned about time etc, then don't waste a single second on this loser, get back out there and find a keeper.
You're 35, the worlds at your feet! I'm 47, didn't have my son till I was almost 40, no reason at all for you to settle. Even then, an anonymous sperm donor would be better than this idiot boyfriend.