I wonder if I am the only one who is feeling sorry for the OP's boss and his wife? The company that @OopsyDaisie works at does not sound like some enormous conglomerate where the only important factors are making money and speed!
The OP's workplace sounds much more like a company my dear departed mum used to work for. It had an owner/managing director, his secretary/PA, and then a few departments that had their own department heads. One was where the manufacturing took place, another was a smallish warehouse, then there was the accounts department - where my mum worked (they still used calculators, and some sort of small machine where you input numbers and they came out on a role, they weren't called computers, but then this was the 1970's) - then the last proper department was the filing department, which was a smallish room filled with rows of files, and just one member of staff who did the full-time filing there. Everyone knew everyone else's names at that firm, and at least a potted history of most of their colleagues.
If the OP's work place is similar in size to the place my mum worked, and just as friendly, then her boss is probably trying to do a nice thing for his staff, and not for networking reasons for goodness sake, just for fun - hence them being invited as families, to just relax and enjoy themselves. His wife may also feel a little stressed about providing the lunch. I know that my mum's old boss's wife used to enjoy holding these occassional events, but as she was a normal human being, she would get a little stressed in case something went wrong! As I mentioned before, they were not some large corporation, who when holding events would employ outside caterers to provide the refreshments etc.
So in my imagination OopsyDaisie's boss and his wife are just trying to do a nice and friendly thing. They would probably be quite upset if they knew that at least some of their invited staff would feel obliged to go, and that some of their staff's spouses and children were trying to think of ways to get out of going! Please go with your children OP, and have an enjoyable time, which by the sounds of it will be much better without your selfish husband's presence.