It seems like there is something in you going on - very deeply and perhaps unconsciously- for you to perform for love.
Paying half the bills to prop up his spending ability for these trips. Like a rescuer - if it wasn't for you he'd never see his children etc
Now you've convinced yourself that being available and wanting his children here is what you want?
You are ignoring the opinions of your adult family and you are torn because you know this whole thing is wrong but you'll be cutting off your own supply nod hard-won worthiness if you ditch him.
I recognise you as partly me and am sorry it's so tough.
You have a great big hole in your defences, no doubt created by your upbringing.
It's all down to boundaries, and you have none here, other than performing harder and harder. Look up Jung shadow integration, and reparenting, maybe someone like Nima Rahmany yt, your adhd brain might enjoy the food for thought.
This isn't a thinking problem it's a feeling one as rationally you know you're being a mug but emotionally you're obviously hooked right in.
Springboard into understanding the roots and then you will have new choices, 4 years in is enough to tell you you're caught up in a puppet dance that Mumsnet won't solve either.
Good luck, it's sure to be tough but the right help will help you get clear x