I've just had a baby and thought I'd feel like you do, however visitors were the best thing ever :)
I stayed in hospital for three days (post c section), which was a lovy snugly time for me and baby, without that snuggling being in the face of my four year old twins. When I came home, I tried to give the twins as much time as possible, but the simple fact is that babies like to be held and fed, predominantly by their mothers.
When grandparents came, there were suddenly more adults to give attention and love. The day they visited I sat on the sofa and held/fed the baby, handing baby to DH or MIL when I wanted to; DH hosted by making tea/sandwiches/cake and got up to date with the washing while he was at it; and the twins had Granny and Grandad to build train tracks/Lego towers/play hide and seek/read with. It was an easy day, because there were two extra adults to share the load, taking child entertainment off DH and I for a few welcome hours.
I am the type of person happy to greet visitors in leggings and a t shirt and expect my visitors and DH to sort everything out themselves. If you would want to be the one tidying up/making tea etc then the visit would be very different and maybe should be avoided unless you can resist your hostess urges!
I guess what in trying to say is that although you have lovely plans of bonding as a family, a few hours when your older DC is distracted by others may be a welcome respite from trying to give the baby 100% of your attention whilst making older DC feel like he has at least 50% of it! It is a pretty hard thing to do!