Everyone is different and it depends, its a how long is a piece of string question really...
Staying in a B&B does help.
My parents now always stay in a hotel (the official reason is because they need to be on the same floor as a bathroom - our house is multilevel and the only bedrooms on the same floor as bathrooms are equipped with mid sleepers or toddler beds which there is no way on earth my parents could get in or out of even if we did want to kick two kids out of their bedrooms for them each to take over a mid sleeper...)
I live a thousand miles from my parents and they stay about a week when they visit - when they stay in a hotel it is OK, although finding things to do is a challenge as neither can walk far, my mum struggles with stairs, and my dad needs to be able to get to a toilet frequently... it does rule out most things that appeal to small children except for things that are dull to my parents (involves them sitting about watching the kids do stuff). I do find them sitting about my living room odd as I feel I am supposed to entertain them, and at home the kids don't want to sit and chat politely or be read to by my mum, they want to go out and play with their friends, which is what they normally do on a home day... Still a week is manageable with parents sleeping elsewhere and me able to drive.
My mum came to stay on her own with us in our house for 2 weeks after I'd had an operation once - a c-section. She came to help but she mainly tried to monopolise the baby whilst not lifting a finger with toddler nor anything household, criticising my friends, complaining about the shower, comparing everything negatively to her home, talking loudly and at length about herself in English to people who speak only rusty school English from 20 years ago (I live abroad) and trying to tell me what I should be doing (telling me I should be getting on with washing floors a week after a c-section was a highlight, as was buying and hanging a wall mirror in the room she was sleeping in) while I had her there "to hold the baby" and demanding I drive her to the shops to buy herself clothes as she felt she hadn't packed for the weather - I wished she wasn't there by day 2 and nearly murdered her on day 3, and I still feel angry about those two weeks and struggle with her more than I did before that stay.
I think them being in your house will be difficult by the sound of it, but 4 days should be about survivable. Not being able to drive or get about much will make it very claustrophobic...