OP, you are in an abusive relationship and because of his emotional abuse, you are now doubting your own reality. He is gaslighting you by turning your very reasonable request (asking him to stop being controlling) and he turns it back on you to make you feel like you are being unreasonable for even suggesting you are entitled to live without fear or control. I fear because of this you have lost your ability to trust yourself which is why you are asking strangers on the internet for validation. I can tell you 100% he is controlling and abusive and 100% that men like this cannot and will not change.
The problem is you are trying to have a normal conversation (you have every right to not live under anyone's control) with someone who doesnt care about how their actions affect you and your child (they have no empathy). This is enough to drive anyone insane and if you are not in a position to leave right now, you could try the 'grey rock' method to limit these cyclical arguments where he will never let you be in the right.
Unfortunately, these relationships have a devastating impact on a child's development (trust me, I have first hand experience) and he could very easily become physically or sexually abusive to you and your child (if he hasnt already). The only way to get this to stop is to leave the relationship. I know this is often difficult but there really is no other option, even if they agree to counselling it is often just to keep you from leaving.
Have you contacted your local womens centre or the national domestic violence helpline (it's open 24/7). You may want to start gathering evidence of his coercive control so that you can go to the police (save texts, emails, recordings, evidence of financial control etc.) which could help you get a restraining order and may help later when finding accommodation (although you could move into a refuge now if you feel in immediate danger) away from him and court cases over contact with your child.
Leaving an abusive man is often the most dangerous time in any abusive relationship so please do not tell him what your plans are or that you are planning to leave him (if this is what you decide to do).
I have walked down this road (left my abusive exH when I had 3 small children) and I know how scary it is but I can honestly say I dont think I would be alive right now if I didn't. Please take care OP, you are stronger than you think