Pickles, this awful bullying man is making you ill.
Anyone with a tiny baby, being bullied into driving long distances, feeling under pressure to leave their baby with a person who cannot be trusted to look after her, would be feeling anxious, depressed and unable to sleep.
This is serious abuse. He is risking your health and that of your dd by insisting on this travelling.
Anyone with a grain of common sense or compassion would not dream of asking someone who had just had a baby to drive 2 and a half hours one way, never mind a round trip.
You should be enjoying this time with your dd. It is enough of a full time job just to look after her, do the shopping, cooking and washing, never mind driving anywhere.
Your depression is not your fault, he is the cause.
Can you talk to your HV?
You are completely within your rights to refuse to do any more travelling.
You need a few weeks of peace and quiet, just you and dd.
Communicate only by email. See a solicitor or CAB.