Rescind your husband's invite. If not outright, at least in your head. He can enjoy his sport with your blessing. What he absolutely doesn't get to to is to come along and make things awkward and uncomfortable for you, or anyone else, by sulking or being 'to the point' (ugh!), making you feel uneasy or rushed at what promises to be a nice social occasion.
Take the children. Whatever you do, make sure they're with you so he has no reason to summon you or get in touch while you're having a lovely time.
I'd second switching your phone off or 'not hearing it'. It's not rude: what is rude is him inserting himself remotely via the phone, as you anticipate he might. I might be way off, but a man like your DH will likely be adept at exerting influence from afar if you pick up; just say a little something that'll take the shine off a lovely time and make you think, as if it was your own idea, that it might be time to wrap up early, even though the sun is shining, the DC are each other's new best friends and you and you're really enjoying the easy flow of conversation and just hanging out. "The chemist shuts at 2 today, can you pick up some Rennies on your way home as you'll be passing that way?"
I notice you enquired about how a PP who left her SAHP spouse is faring now. If you are considering separation, please be aware of how your DH's SAMH status, as default primary carer of your DC, irrespective of how supportive (or not) this role is of family life, will impact whatever settlement you put in place in the event of a divorce. Do you really need him to be a SAHP, especially if he doesn't even seem to enjoy it or pull his weight day-to-day? Is he actually SAHPing by mutual consent or because he feels it doesn't make sense to work / doesn't actually want to work? Should you feel you've come to the end if the line with 'picking your fights' and walking on eggshells, leaving early and doing more than your fair share, you may find that you are in a precarious position. Begin to gather evidence of your suggestions and discussions of him finding employment, of your enquiries into wraparound care for the DC; of your expressed wish that he seek employment and his reluctance to do so. It may come in handy.
Enjoy your lunch with your lovely DC, and best of luck.