I'd say a court order is pointless in your circumstances - as my barrister said to me, there's no court order to make the NRP become a decent parent or even a reasonable human being, and certainly nothing to make them have contact.
I think perhaps your best way forward to to say (by text, email and / or letter) something along the lines of:
This is what we'll be doing between 9 and 11am on Sundays: (eg swimming / soft play / library / at home). If you'd like to see the kids and have them until X pm (maybe 4 or 5?), then come find us there / to my house.
If I don't hear from you by 11am, then I'll assume you won't be seeing them that Sunday, and we'll get on with our day.
If I have particular plans that mean you won't be able to see the DCs, then I'll give you at least four days warning of those plans.
How does that sound? I'm trying to suggest some way that you don't have to get yourself and the DCs geared up to see him, then get let down by him, but that still allows flexibility so that he can come and get them to have time with them without it being at your (in)convenience.