So he is prepared to lose you because he doesn't want a baby, but you aren't prepared to lose him because you do. The balance isn't fair.
I was with some several years ago who had children previously and had a similar battle with an ex to see them. He had also had a vasectomy. He told me that he did want children with me and was keen to get his vasectomy reverses when we were ready. Several years later in our relationship, children and a family started coming up more and more. He became more and more resistant and then said that he had made the decision that he didn't want any more kids. Having a family wasn't something that I had my absolute heart set on at the time and I didn't want to lose him, so I stayed and gave up the possibility of having a family for him. Later it turned out that he just didn't want to have a family with me. I wasn't important enough to him, for him to want to be tied to me for life. He cheated and broke my heart but it was the best thing that ever happened.
I dated again, said if I meet the right person and I am able to have a family, great! If not then that will have to be ok. I would rather be on my own and miss out on that chance, than with someone who is essentially taking that option away from me. I met someone, our relationship flourished, and three years later I am expecting our first child. He has a son already (19 years old). He wasn't sure he definitely wanted children again, until he met me. He wants to have a child that is a little mix of both of us, because he loves me. He is super involved already and we are so excited!!
I came so so close to missing out on this, for someone that wasn't prepared to give me what I needed. I look back and can't believe what I was prepared to give up for him (even though I wasn't convinced that a family was something I wanted).
Have a think, deep deep down, and really consider how important this is for you. Yes, you might not meet someone, and yes you might not be able to have children...but is he worth giving up that chance?
Good luck OP, and sorry if I have rambled on and on. Your post struck a chord with me!