Be that as it may, it is still unreasonable to demand he leave the house every workday at a certain time.
The issues you are experiencing are issues that are 'yours'. Trust me, I understand the claustrophobia. My DH and I are retired. I love him dearly but there are times I think "Will you just get out from under my feet!!!???". He's also naturally a very 'loud' person; needing background noise, humming/singing about the house, 'loud feet', etc. and I tend to like a quiet house and think "Hush. Up!".
I cherish the 'me time' and the silence I have when he goes hiking or 'hobby-ing'. But these are MY issues, not his and I need to be able to deal with it, not demand that he leave the house or STFU. I don't demand he get out of the house or demand X hours of absolute silence. And it's a two way street, I have little habits that irritate him (like tidying up behind him before he's done and hovering interfering when he's doing 'projects') but he doesn't demand I leave the house nor demand I maintain silence. Instead we talk, we explain calmly, and we try to listen and not 'annoy' each other (too much). I support and encourage his hobbies since they serve a purpose for me, too. And he calmly 'redirects my energy' (his term, lol) away from his 'projects' and I try not to clean up things when he's in the midst of them. My point is, it's compromise and working together. And that's worked for us for over 35 years.
If your feelings are that severe and intense, you need to consider counseling or perhaps returning to work. Or something else that gets you out of the house. Because I think part of the problem is not that he's in the house too much, it's that you're in the house too much or have become too 'rigid' in your routines.
I'm not giving your DH a 'pass' on this. He does need to work on his faffing when it affects the family's activities or other people. He does need to pitch in with the DC, especially in the AMs. He absolutely need to show more respect for you and others by being on time. So I'd say counseling for him, too.
It's a good thing to remember that someday, God willing, you and DH will be retired too. If you can't find a way to work together now, how do you think you'll cope when it's pretty much 24/7/365?