My line in the sand has always been that I will not force the kids to go. I will encourage, but I will not force. I will not damage my relationship with the children because ex is such a poor parent that the kids don't want to see him.
Through careful listening, and persuading, and by getting the youngest a phone to use and proving that I will always answer when he calls, and come and get him if he needs me to, I've managed to get them both visiting once a fortnight, although they won't do overnights.
His contact time is for him to manage - do you have a court order now? If not, make the kids available, let him try and persuade them, but make it clear you won't force, and let him take you to court. I'm not saying that you'll win at all, but by putting the onus on him to make the effort, you discover if he is actually prepared to make the effort.
Ditto grandparents. My exMIL doesn't bother to contact the kids, so they don't bother to contact her. My mum and dad do, so they do. Not my responsibility to manage another adults communications.