Hello onmyown. There is lots of good advice here and parallels with my experience. I had all the "lets be friends for the dc's sake etc" but it really translated into "I want to have my cake and eat it". My ex wanted me living around the corner from him in a house I couldn't afford with minimal fiancial support from him. He thought it would be great because he could pop in and see the dcs when he wanted and they could see him they wanted. When I took control and said no, I was unreasonable, nasty, destroying his relationship with the dcs etc. Unbelievable given he left to be with ow.
Every now and then he still petulantly declares I am a terrible mother who doesn't deserve her children. All evidence points to the opposite so I just laugh it off.
Doesn't mean it doesn't hurt that the man I married and loved until months after he walked out can think that of me. But I know it is a childish response to taking the control away from him. And for not ever letting him of the hook by saying what he did was ok.
Thankfully I didn't have to factor manipulative in-laws. I think what I would do in your situation is set some very clear boundaries about when he has the children and when you have them and insist that he makes arrangements for his parents to see them during his time with them. If the timing doesn't work for them your mil must ask you and you decide the terms on which they do it - ie all 4 kids for no more than a day.
And try not to let their attempts to manipulate the kids get to you. Remind your children constantly that you love them but don't be afraid to set boundaries and say no. Easier said than done with 12 year old boys, I know... but children will always prefer to be where they feel safest and most loved, where there is no condition to giving and receiving love. In my experience children feel uncomfortable with the "love me more and I'll give you sweets/xbox/a pony / whatever" routine. Carry on being the best parent you can be with no strings and you'll be fine.
Stay strong lovey - decide how YOU want to live, what YOU want your family life to be like then tell him how its going to be. Don't let him play to the "you're not nice card". I know how hard that is. My exh still has the ability to make me feel very guilty for moving 30 miles away. But then I just look at at my happy settled kids who are doing well at school and have loads of friends and I remember what at fuckwit he is.
so is your ex. Hang in there.