OP please don't do this in you want to stay in this role and progress at this company.
I agree that this could be the very early stages of something that turns into a sex discrimination claim. Keep a diary of what was said, send yourself contemporaneous emails recording conversations. Do what ever you want to do to make sure you have a record.
BUT whilst your manager may be a misogynistic arse, assuming he's an idiot is also a bad idea. Starting to need to document everything in writing is clearly someone who is preparing for this to be escalated and it doesn't take half a brain (particularly if you are in law) to see this as preparation for a sexual discrimination claim.
At the moment you have a manager who raised a concern. Yeah we all know that he wouldn't have said the same thing to a man. Yeah that's shit. But it is also the reality of the world that a lot of us work in. You would be completely in your rights to decide fuck this, I won't stand for it, but time and time again the outcome for the woman is not great.
You have the opportunity to address the concern yourself and in a totally professional manner, without a risk of seeming defensive or that you're going to try to 'pull the discrimination card'. Try that first. Bite your tongue and do what you need to do to show that you are not slacking off, the fact that you have recently pushed a baby out of your foof is not in any way relevant to your ability to contribute to the workplace. Honestly most of the time this works.
This may not work with this company, or with this manager, and you may have to go there, but then if you do escalate, you will have a whole long list of all of the things you have done to show that you absolutely are a top performer and it will be much harder for both your manager and your employer to badge you as 'difficult'. If you start making obvious moves towards a discrimination claim now based on a fairly vague statement (we all know what he meant but he didn't actually say anything that objectively was that bad) you're making it easy for your employer to pick the manager over you, and easy for a narrative to be spun that you blew up an innocuous comment and were trying to cover for your own poor performance.
I have confidence from what you've said you can manage this and come out of it smelling of roses and what will be remembered is a manager who was an arse and how well you dealt with it. I'm sorry that you're in this position, but you can do this.