didn't see a future for our relationship/thought it was a mistake to marry. He then just carried on as normal.
Push-me, pull-you keeps you locked into the relationship and makes you confused about it. Also sounds like a form of gaslighting to me.
If I try and discuss what he has said he turns it around on me and says I am destroying our relationship by casting doubt on it
Ah yes, the old 'blaming' to make her feel guilty technique.
As a result I become more and more apprehensive about asking for help with our baby and end up not telling him stuff
Walking on egg shells.
he called me a "fucking idiot" more than once and seemed more concerned about how it reflected on him
Selfish and nasty.
I have to censor what I tell him or he'll lose patience with me
he still refuses to discuss the other stuff he has previously said
Disengagement. Complete refusal to discuss things with you. Renders you helpless - shadow boxing.
he'll not contact me today to give me a chance to think about how me bringing up stuff from the past is stupid and threatening to ruin our relationship Control by him.
I've picked just some of what you've written OP. This man is emotionally abusing you. I concur with the others - you need to get away from him.