OP...i get the sense from your posts that your exdp is very very immature...and frankly probably not bothered about the children but his mum is , so maybe she is nagging him to have contact so she gets contact.
Do you have an ok relationship with her ???
Right now you need support and help ...you have a toddler and a new baby and you are doing it alone...could you perhaps talk to her ?
I wouldn't necessarily refuse to take the children ( and obviosuly go yourself) to the parents house ...because in years to come you might appreciate the kids being able to go to granny's and you getting a break because i doubt exdp will ever step up to the parenting plate.
I would be stating clearly and once only...you will bring children to the parents house every sunday afternoon for tea or for 2 hrs or whatever SUITS YOU.
You will need a car provided for this ...either he buys you one , or he gives car back .
and when dd and ds are at age 2 and 4 and YOU are happy the enviroment is suitable then and only then will you consider changing access arrangements.
but i would urge you to consider his mother an ally ....she probably loves the kids and shame for them to miss out on that relationship cus esDP is such a twat