How sure are you he didn't tell a pack of lies to his family about the break up?
I was in a relationship for 2 1/2 years and became very good friends with his best friend, only for ex to intentionally stir up trouble with said friends fiancee which he made to look my fault about a year in, to cut off our friendship. It wasn't until the relationship ended i realised that he'd done it on purpose, because he was financially, emotionally and sexually abusive and i had started to confide in his friend. He clearly didn't want his facade of nice guy to be revealed as a sham to all his mates.
I continued to be good friends with one of their other group of friends but he wasnt the type of person you'd confide personal stuff to, so ex let that one be. I was also really quite close with his mum, especially after i lost mine part way into the relationship.
Both stopped talking to me when the relationship ended. It was long distance so i couldnt approach them in person to tell them what had happened, and ex controlled the break up as much as possible by insisting he wanted time to come to terms with it ending (he had forced an engagement on me) before he told friends and family, so made me pretend all was ok when talking to them till he decided to tell them it was over.
From the way things suddenly went silent from them, i am 100% certain (especially since he was a manipulative bastard) that he played the victim, claimed he had no idea why i left him, that it was out of the blue, that i broke his heart etc. In truth, i had been unhappy for a very long time, had told him numerous times he needed to change or i would leave because he was abusing me, and had already had a trial separation in which he immediately reverted back to the sweet, loving guy i first met to get me back, then within an hour of getting back together was controlling and abusive again. It had gotten to the point he was threatening to tie me up and anally rape me because i told him i would never consent to anal sex and he wanted it, as well as only being interested in violent/physically abusive sex, and wanted me to pay for EVERYTHING while he refused to work or meet the requirements to claim job seekers, whilst telling me i was useless, boring, stupid, a burden because of being disabled etc.
The absolute only reason i didnt write to his mum and confess all to her is because i knew she was already miserable. He was the only healthy baby she ever had, with miscarriages and and an autistic son whom had meant she couldnt work again, had no friends, and was isolated, whilst she had also recently found out about her husbands 5 year affair that only ended because they got caught by her husband.
Leaving him wasnt an option as she had been out of work for over 20 years, was primary carer for their autistic son, and him being self employed as a driving instructor (he'd schedule fake lessons to meet his mistress) during the recession there wasnt enough work and they were thousands in debt with less coming in than going out. No way could either have moved out as they couldn't afford where they already lived never mind 2 households. I couldn't bring myself to tell her that her only "healthy" child was a controlling, manipulative, abusive sex offender.
I also know no matter what i would side with my own sister over any other person on the planet, no matter what she had done, and regardless of how close i was ot the other person.
So yeah, whether he has claimed you were controlling and insecure (which it sounds like you could have been from your posts if he was just innocently visiting his ex, especially if he did have a baby with her as suggested by other posters who recognise you) or she has taken his side simply because that is here brother, you set yourself up to lose everything when you dated a friends family member, as unless it all works out long term, it will enver end well.