Hello OP
Thought it might be helpful to put forward a different point of view.
I think it really depends what is happening in the conversations between you. Does he try and encourage you not to see your family when you enjoy it and when they are definitely kind to you, which is controlling?
Or does he simply not want to accompany you, which is fine, he is not doing anything wrong, but it may just not suit you?
The two are different things and it's important to work out whether you are dating a man who is independent and doesn't feel the need to be close to your family (fine) or is controlling (not fine).
What happens exactly when he gives you the silent treatment, is angry/sulky etc? Do you just ask if he can accompany you and he immediately starts behaving like this (not fine), or is there a longer, more prolonged argument where you try to persuade him to attend your family events and make comments about how his past must be the reason why he doesn't want to (still not great that he is sulking, but that context is important).
If he is not controlling, but is just independent, and you want to carry on dating him, then I would just stop explicitly inviting him to family events.
I am saying this because maybe I can provide some perspective.
I was in care and I am not in touch with my biological parents. I have had previous relationships where I have got on famously with my boyfriend's parents and one ex-boyfriend's mum is still in my life to this day (much like an aunt figure).
However unfortunately when I met my current partner's parents, I could tell quite quickly that although they are largely good parents to their son/my partner they are not really particularly interested in being open minded or kind. They absolutely were assessing me, from comments on where I lived, to whether my cousin was going to improve his job, to comments about how I'd 'rejected my parents'. My partner couldn't see their behaviour clearly and couldn't understand why I will only visit them occasionally.
I wasn't particularly interested in creating a rift between him and his parents. They obviously love him but they are the sort of people who are only capable of caring about their own immediate family and have a very fixed view of the world.
My partner and I did have a couple of bad arguments about it where he made comments about how I don't like his parents just because I've had problems with mine and I don't understand how family works - which was very hurtful. I would avoid saying anything like that to your partner.
I have never actually got to the point where I've wanted to say to my partner 'look, I know your parents are great to you, but they're actually repeatedly unkind and belittling to me in lots of small ways, I am afraid they are just not great people so I only want to see them occasionally as I don't want that type of regular small mindedness in my daily life'. I feel like it would be unnecessary and hurtful.
Plus he would probably never believe it - it is far easier to pin the issue on my unusual background. In fact, what my unusual background gives me is a good ability to read people and their motivations and when they are comfortable judging and being slightly unkind to people around them.
I would just be cautious about assuming that your family is the good, kind, happy one and the issue must be with him. If he is being controlling, behaving abusively or making you feel small or scared over this issue, you should end the relationship.
But if he just doesn't want to spend time with your family and you keep challenging him on it and referring to his background in arguements, I think you should rethink your position. He is nearly fifty and not obligated to spend time with people when the experience is making him uncomfortable.
As long as he is not being cruel about that, that is okay. Equally, you are also entitled to leave and find a partner who actively wants to spend time with your mum, that is fine too.