MOS, telling the OP that everyone but two named posters are "the Jeremy Kyle brigade" and to ignore them, in a post beginning, "Oh Christ" is not offensive?
OP, I hate to say this, but this isn't really about you now - it's about your child. She will suffer if you are impoverished, hence the law's focus on a fair share of the family money following the children, and he has made it extremely apparent that he can't be relied upon to think about her best interests. He obviously knows that he can bully you into giving up on money your child will need in the future, and he's selfish enough to want to do that so he can live more comfortably at everyone else's cost.
The simple reality is that a court will look very askance at anyone refusing to go to mediation, and his solicitor will tell him as much. He's relying on you being too exhausted and miserable to tell him where to stick his lousy offer. If you've had legal advice, then the simplest and cheapest thing to do is to email him, stating that you intend to attend mediation to resolve the financial impasse and that you therefore need him to provide dates that would suit him so you can move forward. Judges don't like it when people refuse mediation for no reason, because it's the lowest-cost and lowest-conflict way of moving forward. He won't be able to bully you in mediation, because the mediator will be there. And apart from anything else, emailing him in that way would show him you mean business.
Approach it in bite sized pieces - don't think too far ahead. Email him to agree to mediation. Don't think past that email. If he agrees, then you can work on what you need to achieve in mediation. If he doesn't agree, start court proceedings... but just think of it as paperwork to complete, don't think beyond that stage until you need to. Just don't think too many steps ahead, because I suspect it's the whole thing that is making you feel so overwhelmed - look at it as one stage at a time, don't think past the step you need to do in the here and now. The fact is that your situation is quite simple, and judges will see it all the time. The idea of COURT is really scary, but the reality is, a family court is just a bunch of people whose job is to try to get a fair outcome for everyone, as far as they can. That's what they are there for - to help people in your sort of situation. You don't need to do any "fighting" because from what you've said here, your situation isn't complicated, and the decision on outcome is the judge's. You could very well end up with better than 50% - he could be told you can stay in the house until your child leaves full-time school, and only then can he get his 50% out, for example. There are a lot of things that could happen to your advantage, if you just take a deep breath and take this one step at a time. You just need to be honest and explain. And the sooner you send that email about mediation, the closer you are to it all being over and done with.
Do you have a friend or relative who is good at paperwork type stuff, academic? Because there's nothing to stop you asking them to take over the practical side. You'd have to go to mediation, and if things got to that court, but if say your Mum or Dad are willing and able, then just get them to do the practical side for you.
There are ways around this feeling so overwhelming. Really, there are. Your baby won't always be this small and things won't always feel this bleak, and the sooner you get your money and get rid of him, the sooner the dark cloud will lift.
And as to his saying he won't look after her every other weekend unless you give him 20% of the assets... what guarantee do you have that he'll keep doing it, even if you do? I do hope you've contacted the CSA already. That money should be going into your account.
Hang on in there. This will be one of the worst times of your life, but in a year, you'll be looking back from a far better place. Are you posting on the relationships thread? Legal is the place for legal advice, but they are invaluable for moral support, IMO. I think you need both.