With the best will in the world, OP, you sound very young and perhaps with a rose-tinted specs view of how your relationship with your child's father ought to be. You are also probably still quite hormonal from pregnancy, birth and the whole PFB syndrome that sends us all a bit crazy until we learn to play nicely with the wider world...
Having said that, though... my GOD, but you need to get a fucking grip! Your baby is not his father's precious first born. That would be one of those 3 year old twins you resent him spending quality time with. Why? Because they don't miss out, because they have their Mum...? Well, guess what: your baby has you. And at 4 months old, believe me, he couldn't give two hoots about whether his father is with him or being drowned in the hell of a splash pool with his older siblings! All your son wants/needs right now, is a mother who behaves like one, rather than a jealous schoolgirl.
I can see/understand both sides to this. My own son was just a year or so older than those twins when his father left for another woman with their newborn baby. He hasn't seen his father alone in 16 years now (he's 18), because the OW was younger, extremely jealous of my ex's time and insisted that she and the baby had to be included in every activity that was planned during the 18 hours a month that my ex would agree to have our son. Even when they weren't suitable. It didn't make any of them remotely happy, my child has a fractured relationship with his father - which my ex now bitterly regrets, but it's too little, too late - and my son might love his two younger siblings (15 and 12) but he has little to no relationship with them. All he wanted was for his Dad to value him enough to take him swimming without his stepmother, small children who demanded all of my ex's attention and for his father to teach him how to do things. My ex lived with the OW and their kids, so it wasn't like it was two access visits combined. My son has missed out because his father is nothing like your son's Dad. Your boyfriend is actually being a decent parent by valuing all of his children. Maybe you ought to be encouraging that because, sadly, I suspect that in a year or so, your son will be the subject of another Baby Mama's lament about their child being the one "left out". Partly because your jealous insecurity will drive your boyfriend away.
On the other side, my 27 year old daughter's partner has a 4 year old from a previous relationship - and she's having to learn to deal with the fact that he is a decent, involved father who is fighting her over "being allowed" (by her) to spend time with his child in the home that they share. She resents that any child they might have together will not be a shared "precious first" - because her partner's been there; done that already... even though she watched her own little brother endure a neglectful father (so did she, but she was older). I've lost count of the times I've had to point out to her that her stepchild is 4 years old and is worth having a father who loves them enough to want to put them first... even over her jealous insecurities. I applaud my future son-in-law for that, but I also can recognise the wedge that she is starting to drive between them.
My ex isn't happy, my son is very resentful, and my daughter's headed the same way, sadly (except her partner will choose his child over her, because he's an excellent father!). Don't turn that into your future - if only for the sake of your baby and his older siblings. But above all, let go of the jealousy. It's not a good look, not even on an anonymous forum...