"He's not on the BC for many reasons and threats he made when I was pregnant and newly postpartum."
@LD233 - unfortunately, this alone should have been what told you how this situation was going to pan out. I know it's hard, especially with a young baby and you feel like the other parent isn't doing their fair share (I'm a single mother, too). But any 'father' who threatens the mother of his child whilst she's pregnant or newly given birth, ie, when you were literally at your most vulnerable... is not a decent man, never mind a potentially fantastic role-model for your baby.
If he doesn't contact you, how is contact arranged? Does he just turn up on the doorstep unannounced and uninvited? Because surely that could be construed as harassment...? What would happen if you just stopped answering the door if you know it's him on the other side? Could you invest in a Ring doorbell, if you don't already have one?
He's not interested. Your baby is too young to tell you if they are being neglected or passed off onto their 'stepmother' for care (which... I'd say is 99.9% recurring, likely to be happening, I'm afraid). Your child deserves better than this.
Deep down, I think you suspect (and are probably right) that he was only filling in time with you, until he could talk his way back into his ex's bed/home/life. This isn't about the older child, or your baby. This is about him behaving appallingly to you - and to her. Like others have said, the ex won't be too thrilled that he's procreated elsewhere, because how is she meant to explain that yes, her son is four, and they live as a family with the father... but the baby/toddler's not hers, but some other woman's...? You were, and I am truly sorry about this, a protracted affair of sorts. If you're pushing for him to do more for your child by him - which you already know he's not interested in, so it simply isn't going to happen (which yes, is shit and horrible for your baby as they get older), how much of that is because on some level you're hoping that he'll leave the woman he went back to, for you? He won't. Because he's got what and who he wanted all along. Had his cake... and ate it, too.
Is this the sort of life that you want for your son? The example of what being a man and a father is all about? Because as he grows up, your son is going to feel very misplaced in his father's family unit, especially if there are more children born into it. That could happen. That would be extremely confusing to a young child - and cause emotional and mental damage to them. If, as you say, you've had issues with the stepmother, too... then she's going to really resent having to care for your baby with her partner. I'm sorry, because it really is a horrible situation, and unless you're prepared for this man to yo-yo between the two of you (what happens if they have a row, would you let him back into your bed...? Then you row, and she lets him back into hers...?) and make your child confused, less-than, unwanted and all the things we try to avoid exposing our babies and children to, as mothers, you need to understand that you need to either accept the one day a week and stop pushing for further contact - because you're actually allowing this man's ego to inflate, because you are chasing him... albeit, in your mind, on behalf of your son - or stop contact altogether.
If he takes you to court, and they insist on a weeknight, calls during the week, and so on... they cannot enforce that. So chances are, you'll be worse off. Your baby will feed off your stress, and there may even be some abandonment issues already lurking.
Make friends with other single parents. Support is out there, from people who're also left holding the baby. This is hard, and upsetting - but very doable.
Advocate for your baby.
They, and you, deserve so much fucking better than this sorry excuse of a man/father. Maybe set your bar a little higher, too, next time.
But stop giving him the ego boosts, because regardless of your feelings for your baby's father... he really isn't interested in either you, or your baby. 