MardyMarie I am sorry you are in this situation. I think it sounds like a very complicated one. I wonder why you had four children in six years and now feel upset you never get any time to yourself. Are you sure you are not sending out resentful signals to the kids about him? Did you both choose to have four kids in such a short space of time? Might he resent being dad to four such young kids, or was he on board with this?
"They all refuse to be alone or go out alone with him..." I find their dislike of their father very odd. I am very sorry to say this but are you sure there is nothing untoward that had happened that has upset them? Nothing inappropriate, any hitting or anything else?
I think you will have to choose your children but if you want your family to work but he will always be there dad. Is he going to leave you and support financially but not see the kids?
How is living apart not working? is it making things worse? I am not sure how things could be worse than the kids not wanting to be with him at all.
Personally, I would investigate family therapy and a shared parenting course. I'd go for something like Family Links Nurturing Course/Programme, familylinks.org.uk/the-nurturing-programme
There may be a creche facility and you and your dh would need to do the course together, I think, to get maximum benefit from it. You may be able to sign up to this for free in your local area, or you may need to pay.
You said "He loves them and wants to see them..." but you also said "...he's their father and I'm trying to help them have a relationship but I'm not sure he would at all if I called time on our relationship."
So it is very confusing how much he is actually committed to them.
Does he just tolerate them because they come with you as a package? If they have worked this out, I can see why they would not want to be around him.
"...and all 3 older ones have asked separately that he doesn't come." I am very surprised that the two year old was able to articulate to you that they did not want him to come to the farm visit. Could the older ones have put the two year old up to this?
Basically, he can't be there friend, because he is not a boy or girl of 6, 4 or 2. He is an adult man so his role here as their dad is really is either dad or nothing much. The lack of boundaries etc means he is abdicating the responsibilities of dad. Strangely, when I am strict with my dc, they respond better.
I think you and your dh need to work on this together, whether you stay together or not, you need to find a way to fix this, if you can. Good luck.
PS if any questions or comments are too personal, please do not feel you need to reply or respond, I am not trying to upset you, or probe too much. I am just genuinely confused that this situation has come about and it seems to have all hit you now. Maybe there was a light bulb moment.