DHs dad stopped seeing them much because their mum told them he didn't want them. Then refused to help make it possible for the 3 of them to go and see him.
She would tell him she'd drive them to him. Tell the kids he had cancelled.
She would tell him the kids had said they didn't want to go. Tell the kids dad had gone out with new wife and her kids.
When DH was early teens she'd drive him to the edge of town and drop him off, 8 miles from his dad's. Tell his dad that she would deliver him to the door.
In between times, whenever she had a new man, she would drive all 3 to their dad's and drop them at the end of the road on Friday evening or early on Saturday morning. She wouldn't tell him they were coming, or collect them again...
Oh, and she was the one who had numerous affairs and then threw him out because he was angry when he found out! She told all 3 kids their dad was a violent man... DH was old enough to know that she was the one who threw things, lashed out.
It isn't always the man's fault. DHs DD never gave up trying to have more contact. Always made room for them in his life, always made sure they knew where he was, phone numbers, bus routes. Always paid for them (way before CSA etc) and sent cards and presents.
Sadly DHs 2 sibs have never had got over what her mum told them as kids. She did try to put it right when they were in their 20s/30s. But it was too late for them to change their minds. So much so that when she died BIL and SIL told everyone they were now orphans!
So, in my experience it really isn't always the man's fault when contact is broken. Sometimes a bitter mum can be the root cause.