OP please please don't let this man back in.
It sounds like you and your ex coparent effectively and maintain a good relationship, based on the fact you were willing to move closer and the fact he put consistency for your son above all that.
This is really positive. Having loving parents who are on the same page (even if not together) is such a strong protective factor for so many issues.
Then playing games together sounds lovely. I do the same with my siblings. We play while chatting. It's nice because they're doing an activity they enjoy together while catching up. It's likely, especially as he goes into teenage years, that your son will use these opportunities to speak with his dad about difficult or embarrassing topics (in the same way some find being in the car is less pressured). I'd really push to continue this. It's easy, simple, quality time that can be done from anywhere.
It's so good they have that and I'm so upset thinking of your son believing his dad stopped it.
I'm glad you kicked your boyfriend out but I'm concerned that you're posting on here and therefore perhaps having doubts. I really am not one to shout "LTB" but this new man is treating your son appallingly.
Under his roof? It's quite literally your roof, for a start. But also it's your child and your parenting. All parenting issues are between you and the child's dad. He is completely overstepping by going against what you have decided and thinking it's appropriate to punish your son against your wishes.
Frankly, if the way you parent does not align with his expectations then his only choice is to leave. He does not get to overrule you.
How dare he? I'm actually horrified that he already feels comfortable enough to do this and it is 100% going to get worse (trust me).
As this gets worse your relationship with your son will worsen. Protect him from this man. There is nothing good down this road and your son needs his mum to protect him from people like this.
Do not let this man back in your lives in any way. Show your son that you will always have his back and never let a boyfriend treat him that way. Show him, through your actions, what is acceptable and unacceptable in a relationship.
He is at such an important age. There is plenty of time in future for you to find a new relationship. Your son sounds like he has a really good family situation at the moment and a strong relationship with each of his parents.
Please don't throw it away for a man who you have known for such a short time who removes your sons access to his own father against your explicit wishes in your own home.