If the ex is difficult and demanding, and you get serious about this guy, your relationship with him will be a lot of hard work. It will be much, much harder work than a serious relationship with a childless man (assuming it is non-abusive).
Ex-partners who are the resident parent, even in 50/50 residency setups IME, will call a lot of shots in your life. I've had it easier than a lot of step-parents on this board, I think in part because my DSD lives with us for a full week every other week, and we have been able to establish boundaries and a good relationship through that schedule. But even then, the decisions DSD's mother made years ago - decisions about where to live and where to register DSD for school - limit what DH and I can do with our own adult lives. We cannot move city to take a better job. We cannot move across town to access a better school district for our own DD. Unless we want to reduce DH's access to DSD. It's not usually something I dwell on, but you'd be surprised how bitter you can feel when what you want for your life becomes limited by someone you have nothing to do with.
And another case study/warning: a friend of mine, who was living with a man with two children, was recently dumped. This woman knocked herself out to build a relationship with his two kids, establish boundaries that everyone could accept, and acted as a stand-in mother for two kids who really badly needed some parenting. When they were together, they were a family. But her DP's ex interfered a lot, and demanded my friend and her DP do things her way, a lot. And her DP was reluctant to stand up to his ex, lest he lose access to the kids. So, they fought about his ex, a lot. My friend was dumped in her mid-30s after five years of a serious relationship with this guy, because he felt it was easier to start over with casual dating than to rock the boat with his ex.