I expect that's what my my ex might say about me.
Of course he was doing damn all for the kids, or any household requirements (ie. anything from bills to tax returns), and, when I was heavily pregnant, he'd made some comment about my hairy legs when he was basically covered in hair from head to toe, so I'd said if he just spent a month removing all his body hair like he expected me to do, I'd go back to it - and he couldn't be bothered to, so I enjoyed the freedom he enjoyed too. Bear in mind, my body hair was nothing compared to his (which not only affected his body, but the amount of shedding and therefore hoovering was considerable)
He could have been up front so I knew to end it. He could have ended it himself. Instead he decided to ignore me and the kids, 'work away' and get up to allsorts in male saunas, with prostitutes, and with women he met in clubs, until I had what was sufficient evidence to end it myself (because I'm not a jerk)
So YABU. Grow up and be up front - end it, tell them why (even though I'm sure the reason you don't want to is it seems petty).
Cheating is never acceptable.