You don't need his agreement to take your child.
He doesn’t need the OP’s agreement to take their child to stay with a relative either.
OP it sounds like the best thing to do right now is to speak to him, try and agree that your child goes and stays with your parents for a few hours or a day so you can both talk about the next steps. It might well be that neither of you move out right now, if he has nowhere to go and you don’t want to go to your parents without your child (wouldn’t it be best for the child to remain in their home for now? Less upheaval?).
Explain you know it’s over and think it’s for the best and need to decide which of you is going to remain in the property and who will move out, How you’ll split custody etc. Potentially agree that while it’s all still fresh one of you will sleep in another room to the couple bedroom for a few nights or so so you can both cool down and start thinking practicalities.
You say you don’t want to go because that gives him power then to be the one to remain in the property, I don’t understand that? You said it’s rented, aren’t you both on the tenancy? If so then it doesn’t matter if you stay elsewhere for a while, you’re still legally as entitled to live there as he is. If it’s in one person’s name then that person is the one to remain and you should be aware that they can kick the other out at any time.
You said you can’t afford to live there anywhere so surely if he wants to stay there it’s better for both of you, he has a place to stay, and you can find somewhere you can afford or move in with parents for a bit.
Whatever you decide, in this case it is probably best for your child to remain in the family home with whichever parent is there as it’ll be less stressful and upsetting than having to move out to your parents out of the blue.
I understand neither of you want to be apart from the child hence my advice that you try and stay in the same house for a bit in separate rooms to cool down and start figuring out how to split.