Long post sorry, bear with me.
I was also in a relationship like this in my early twenties. It was very similar except my now ex and his ex had to go to the cinema together every week by themselves because "it was tradition" or some crap like that.
I was absolutely fine with it as I have/had no trust issues and knew we had an amazing sex life. However I had an older female friend at the time who made it clear this was not normal behaviour. I happily continued in my delusional world until his birthday rolled along.
I arranged to take the day off work to spend with him and asked what he wanted to do. This was a few weeks before his birthday. He said he'd think about it and let me know. Then a week or so before his birthday, after his regular cinema day with his ex, he sheepishly came to me at work (we all worked together too which didn't help 😂) and said he had something to tell me. Apparently his ex had arranged a surprise birthday party for him, at his flat, and had invited all of his friends. I said: "Ok, well, I didn't know about that but maybe we can go to it after we've spent the day together." And then he said: Errr...well...my ex said you're not invited because you're not fun." 😂😂😂
I laugh at it now because what a bloody ridiculous thing to say. The girl had never spoken a single word to me, or smiled at me. She didn't work at our workplace regularly but I still remember holding the door open for her once and she breezily glided through and blanked me. I remember thinking that was odd but thought nothing further.
What a sad case. She was clearly a few sandwiches short of a picnic to do all this.
I flew into a rage and quickly dumped him. A week or so after he came crawling back, apologising, but it still took him another few weeks after that before he told his ex all the secret cinema trips were off & he couldn't see her again. He didn't do the birthday party, I actually sat outside his house to check like a stalker as I had the day off work anyway & saw her storming off from his house in a huff & there was definitely no house party going on. We dated for at least another 18 months I think but guess what eventually broke us up? His ex, moving in with him!!! 😂🤦🏻♀️ He lived in a shared house (all males) and one of the guys was leaving and so they were looking for a new housemate. His ex had kept in touch with them all and accepted the invite. I couldn't deal with this madness so broke up with him which was really hard as I was in love with him.
My older friend at the time said something that's always stuck with me. She said he was allowing her or maybe both of them to have a fantasy relationship, because in her mind, she's still believing and acting like they're together. Friend felt vindicated when all this birthday crap happened and then when the ex had the gall to move into his house. And I hate to admit it but she was right.
They don't have to be sleeping together/having sex to still have an intimate bond and be fuelling this by refusing to allow new partners into their secret little world. It is unhealthy and I don't know how old you are OP but I'd honestly dump him and find a man who doesn't need to have his ex constantly hanging around like a bad smell.
The dirty looks I can relate to and aren't a good sign 🤷🏻♀️ Good luck OP, you can free yourself of this anxiety you're experiencing and find a good man 🙏🏼 Xx