I would walk away, its never going to change. It will only change when his ex deems it to change. Your relationship is at the behest of someone else. You either accept that and let it go on, or you walk away.
A relationship is meant to enrich your life; not cause you sadness, pain and stress. The other person in that relationship is meant to cherish you and not cause you pain by their actions or lack of action - he can't because it may impact his relationship with his children; so instead of owning that inability he blames you, because thats easier - you're being too sensitive, this is normal, you're being unreasonable, you're being a brat.
You're not being a brat - if I was bothered by something my husband was doing or involved in, and I took it to my husband, his first goto wouldnt be what is wrong with you again this time? are you still going on about this, ive told you before this is normal... my husband would be "why does it bother you? what can I do?, how can we sort this?"
You can end a relationship for any reason, and being unhappy with how he acts, or how he reacts to your unhappiness is a good reason. You cant control the actions of others (so you cant say - change how talk to her, don't take as many calls from her or do xyz) but you can control your reaction.