OP, beware listening to Mumsnet too closely on this issue!
Mumsnet is full of some very anti-social people who don't basically don't much like having other people round at all.
Have a look at a thread on having people to stay over. You'll get loads of people saying they hate hosting others, don't feel they should, why can't visiting family stay in a hotel etc etc. I don't know anyone like this in real life, but I suspect the nature of this, an online forum, means you're less likely to get those busy socialising as they're often off out being busy with other people not posting on a forum!
And asking if you're insured?! Give me strength.
Also there's a lot of unconscious sexism, lots of thinking if the poor husband and ignoring the nerds of mothers, we're socialised to out others first and that really comes over IMO.
If it suits you to have the local kids over and you feel it benefits your DC, please don't be put off.
Yes, it's wise not to get into a position where everyone expects it and you feel you can't say no, but if you want to in the daytime,why not, as long as neighbours aren't upset by it. Do you know them well enough to ask?
As for when your DH gets home I think he's being entirely unreasonable if he expects to never see other DC when he gets home. Part of being a parent is dealing with other people's DC, and putting your DC first!
If he wants some peace and quiet and family time that's nice, and I'm sure you can come to a compromise where this is happening on a regular basis, but if what he wants is for you to deal with things related to children and him to be lord of the household where his needs are put above others,he need to get with the times!