I read an interesting article a while back on the topic of fathers who drift out, it related particularly to fathers who were typically poor and relatively uneducated, who often had shortish (under a year to maybe two years at most), live in relationships with women, who themselves typically had kids with several different fathers.
The article suggested that a fairly typical pattern was that they would be initially enthusiastic about being fathers, but it would start to fall apart as the relationship began to break down with the mother. Many of them struggled to provide financially, and didn't have good examples of fatherhood in their own lives. Once they had moved out, all kinds of other barriers asserted themselves - the kids were still small, they often did not have cash to take the kids out or a home where they could visit, they not uncommonly would soon move in with another woman and her kids and the mother would often have a new boyfriend. Friction between the parents tended to make them argue over things like visiting arrangements and the mothers were upset over lack of financial support.
But as a result of all of this it was difficult for the fathers to build a strong relationship, even from early on, and whatever efforts they made in many cases still didn't work out well - the children weren't all that happy to see them or didn't enjoy the visits. Eventually the visits would drop off.
Not all of these things apply more generally, but I do think there is something common to the experience of a parent who doesn't have custody, even partially or at least fairly often, finding that the basic relationship of trust and comfort that a child normally has with a parent doesn't really establish itself. And things like friction with the mother can of course apply anywhere. So the whole thing becomes fraught and difficult, and the children themselves don't function comfortably in the parental interactions.
In general it is much easier for parents to both remain close to the kids if they have stable lives - housing and jobs - that are not too far apart from each other geographically.