I've been here, and although with hindsight I can say that what everyone above has said is true, when you're in a relationship like that, it's tantamount to being emotionally, mentally, sometimes physically and financially abused.
I don't say it often, but LTB. I would consider your future very very carefully. I chose to split with my dope-smoking and heavy drinking wanker XP in the end after he stayed out on three separate occasions and switched his phone off. He was obviously sleeping with someone from work too, who was herself a complete mess. He was and still is a manchild.
He first insisted on 50/50 custody (fine) but because he didn't drive, me (and eventually my new partner, who became wonderful, sane, caring, brilliant, won't-take-any-bullshit DH), had to do the school runs on his days of access, pay for 100% of DS nursery fees, and continually replace lost school equipment and uniform.
As they grew, and school demands grew, I was never listened to regarding attaining stability for the children - living in one house one day and another the next was exhausting for them, but he wouldn't hear of them spending one week at each house, (because it didn't suit his lifestyle or convenience), so eventually, when I said that for the sake of DD's GCSEs, they ought to have one main home, and one secondary one, they chose their dad's (after years of him whispering in their ear, monitoring their texts, and asking them about what went on here).
Eventually, DS decided he wanted to live at his dad's - after his dad had had a mental break episode - and we didn't see DS` for a year but DD came around every other weekend. This has almost killed me emotionally over the years. Things are better now - DD is at uni (far, far away from him, thank god) and DS is 18 now, and visiting every other weekend and half the holidays - but it damaged my relationship with them, and that of their younger siblings, stepdad and my parents, forever. We know they're not at fault, but when people do a lot of weed, they often get paranoid, they sometimes develop mental health issues, and that has ramifications.
If your P can't sort himself out, then prepare yourself for a very long and horrible ride until they turn 18. I wish I had done what I'd wanted to do at the time, and move with the dcs somewhere very far away and stopped all contact. But I tried to do the honourable thing in enabling them to have contact with their df. I know it's not the pc thing to say, but it wasn't like dealing with a sane, normal, logical person. If he's not living up to his parental responsibilties now, he's not going to in the future, so you'll probably all be better off if you cut him out of the picture.