VB - this is all horrible for you 
For his parents to say you do too much and they would rather he flounder, is a massive thing. Most parents would back their son, see no wrong in him and some would even blame you. This must tell you all you need to know about him.
You really do need to detach. I understand about the work situation, but this is so confusing. He left for OW, because he wanted to. He should be happy and she should be the one supporting him. A happy person would not need you to prop him up, he would be getting enough from her. Obviously this relationship isn't the love story he thought it would be.
But this is none of your problem. Whilst I appreciate your need to "shore him up", I do think you need to gradually detach more.
It is confusing for the kids to see their dad come and go to the family home but not stay there. Especially as he makes himself at home. This I think is the first thing you need to tackle. Arranged times for him to see the kids. At a neutral place perhaps - could be his parents? Or this time of the year, meet up in the park. You don't have to do this all the time, but to start with gradually introduce a few meetings at a neutral setting. This will be good for the kids, as it's less confusing, and good for you. Most importantly it's placing boundaries for him - he is the father of the kids - he can see them. He can't though continue waltzing whenever it suits him, become dad at home, then go home.
Does he have keys to your house? If so, I would get the locks changed. Keep the doors locked at all times, and if he turns up unannounced and you don't feel strong enough to tell him that it's inconvenient, then ignore him. Make it clear that you love him seeing the kids - but it has to be at agreed times - you will meet him with kids - and then collect kids from him. You can then do your own thing, and he will get the message.
Right now, he holds all the cards - emotionally - he's playing the vulnerable card. Financially - if he gets upset he might now pay up. This has to change. Do it slowly, but please formulate a plan - because otherwise this could go on forever, regardless of whether he gets a house or not.
Be sure that things aren't great with the OW - if he was getting all he needed from her, he wouldn't need you so much. But that's his problem. He made his bed, he can lie in it - (and not any of yours anymore!).