I think he is being confusing. I don't think you are doing anything wrong at all.
I do think you need to have words with him about whether he is up for another child or not.
I agree that some men are afraid to commit/admit to TTC and prefer it sprung on them, when they have not exactly been careful, but some are actually really awful if they have it 'sprung' on them and even if it isn't and they have agreed to more babies.
I had one baby with a man who agreed to a baby, and then changed his mind/freaked out and used the fact that I wanted to keep it, to be really nasty and controlling toward me on the grounds that he was doing me a favour letting me keep it. I binned him in the end of course but the baby was lovely . This was my failed attempt to get it right, following the previous disastrous set up:
I had my other (before this) in altogether more shady circumstances - he definitely didn't want any, and was using me, and was drunk every time he slept with me, and often took risks, and one night I didn't take proper precautions and he didn't ask me to. He was furious and insisted it was my fault but I kept the baby and he stuck around on and off, still using me.
It was a ridiculous 'relationship' as neither of us was honest with the other, we both wanted different things, and the baby I had is now a child who has never known his father, which is just bloody awful and I live with the guilt every day. I was very stupid in those days.
Do be careful before entering into a situation where you are not perfectly clear with each other, as it could create a lot of problems.