I think this is the crux of the issue. Being useful to the family has become your job to a certain extent, and maybe your sense of worth is very dependent on that too. "The Family" has become everything to you, and if you don't have any interests outside of that then any changes within The Family probably feels like your reason for being is no longer assured. Maybe it's a subconscious thing.
My son is also an only child, also IVF and he started uni last september, so I understand where you're coming from to a certain extent. I did miss him a lot to start with and not seeing him for 8 weeks felt really odd. I'm a single mum and we are close, so there was a real mix of pride and pleasure that he'd got into a top uni and that he was making friends and settling well, but also a bit of sadness that he seemed to be so happy being away from me 😂
These feeling soon passed though, and I've realised that him being away and us going days without speaking has absolutely no bearing on how much he loves or values me. He's just older, and living his life and doing young man stuff, which is absolutely the way it should be. I wouldn't expect him to continue to always be available for a family tradition now he has a life elsewhere, even more so if there was a girlfriend on the scene.
I think you need to accept that a regular family get-together that involves so many people, especially young people, is never going to carry on without any changes, and it's unreasonable to expect it to. Those young cousins of his are quite likely to want to start doing their own thing in the future as well, but it's not something you can control, and its not a refection on your worth or usefulness, or how much everyone means to each other.
In the long run, it really is best not to expect your son to stick to a tradition that doesn't suit his life now, and as others have said, making an issue of it will go against you. Let him and his girlfriend know you're always there and will always enjoy their visits,but those visits can be when it suits them as well as you.
I think it might be worth you starting to think about what life outside caring for the family might look like? I don't mean huge changes, as change can be scary, but what about something like a bit of volunteer work? That way, you get to do as much or as little as you like, but you're still doing something useful that's not solely tied to family.