I think you should do exactly what you want. Having your first child is overwhelming, the birth may be long/tiring/complicated, you may have a CS, take a while to figure out feeding etc etc. I didn't see anyone for a full three weeks -- partly mandated by the fact that all immediate family is overseas, and even if not actually staying with us, would have had to be picked up at the airport, fed, ushered around central London etc (elderly, not very city-savvy parents with zero sense of direction, and a terror of the tube). It seemed easier to put visits off until I was feeling up to it, because it would have been very full on.
And in the event it was lucky we'd said three weeks, because I had a CS, lots of blood loss, milk didn't come in, DS had jaundice and lost a lot of weight, was readmitted etc etc. It was completely hellish, HV, and La Leche League, lactation consultants, GP were no help and DS never stopped crying. It would have been ten times worse with well-meaning people who last parented in the early 70s added to the mix.
I'm sure our families would have liked to see DS sooner (he's my parents' only grandchild, and the IL's next youngest grandchild was 14), but I decided that didn't trump my desire for some time with just the three of us, and no one was rude enough to mention it.