You rarely hear of men stopping access to the mother, it's always the mum trying to control the dad by refusing access.
You only have to have a quick glance on MN to see there are a whole load of deadbeats dad's out there. Even the ones still married to the kids mum are pretty useless. But there's also a load of good dad's out there in loads of debt due to custody battles because their ex is using the kids a weapons. They can be thousands in debt, having to pay full maintenance because the ex won't let them see their kids and eventually the money runs out without custody being sorted. Like you say, it's no wonder the suicide rate is so high.
We have my DSD 50% (on paper) but her mum often asks us to have her extra days. Generally we will say yes as we'd rather DSD was with us than left with some random relative. However, my DH has always checked with me if we have something on before agreeing to have her. If we had a date night scheduled or whatever then the ex would be told no.
My DSS moved in with us fulltime a few months ago. Again, this was something DH and I discussed and I fully supported. DSS is now 17 and working so we also discussed expectations for him living here.
My DH appreciates that although the kids are a priority, he must prioritise his marriage too. I do a lot for my family, I do most of the school runs, take DSD to her hobbies and arrange playdates. I'm happy to do this because I feel appreciated and listened to in my home. There's a lot of posts on the step-parenting board where SMs are pulling back because they feel taken for granted and like they're being treated like a free nanny and housekeeper.
Ultimately, kids will grow up and fly the nest so if you haven't made the effort to nourish your marriage it's only a matter of time before that falls apart. So yes, while kids should be a priority, so should the person you're going to spend the rest of your life with long after the kids have moved on.