Like you OP I have anxieties about being a complete family because I didn't have that growing up. I have shared this with my DH, and you know what, when I get anxious he hugs me, calmly sits me down and repeats over and over "I 'm not going anywhere, I married you, for me that means life". And so over time I am getting less anxious because he is helping me with my anxiety not feeding it.
This man is a nasty, nasty bastard. He is using your understandable anxiety that you have shared with him out of love and trust to manipulate you. He is trying to get you to "agree" to rape in advance and play along with it in costume.
I agree that he wants a way of not feeling guilty- either when he forces you into sex at a time when any decent man would back off, or if he leaves he will be able to say "your choice, your fault". This is a further sign of his conniving abusiveness not "choice".
As many pop have said, a decent guy will understand, recognize, work out for himself when is a good time for sex and when isn't. Whether that is around pregnancy, illness, past trauma. We went through a spell of nearly two years of no sex when I was going through counseling for a past sexual assault. He never once complained and even gently steered us in another direction when I tried to initiate because I felt guilty because he knew it wasn't;t the best thing for me, or for us in the long term.
I really, really believe that abuse is the only good reason to end a partnership that involves children. There is so much abuse here that I really feel you owe it to both yourself and your children to just cut this relationship stone dead.
Good luck and take care of yourself.