Our LO has been looked after my MIL for the afternoon I went into work - he was absolutely fine. I think for us the attachment is clear when we're in the room, but when we're not there he doesn't particularly need us so long as he's entertained. If he needs something like food or is tired, he will cry for us, and if one of us is in another room where he knows that we are (if one is cooking) he has to continuosly check to make sure we're still there!!
We also went out for dinner, but after we'd put him to bed, so he didn't actually know we'd gone out if that makes sense! My parents stayed in the house for that, but didn't actually have to do anything!
We stayed over at grandparents slightly too early on I would say and it was pretty horrendous (about a month in). He had a cold and it was just a bad idea when he was under the weather, and he was up every hour and only slept when being held...! Though I think this helped for attachment purposes...! I think if your LO is a good sleeper I would do it - we stayed over at a later date at Christmas and it was wonderful being able to celebrate with everyone. It was hard work (you don't know where anything is and people don't understand the routine, and some people are too loud and exciting close to bed time...!) but we all slept really well. If there's other children, other people will be up at a similar time to you (depending on how early you get up, we are up at 5:30-6...) so they will understand when you say you're going to bed at 9!
I think it's natural to feel apprehensive but at the same time I definitely think my KIT day made me a better parent and to be honest I really doubt he remembers that he was ever looked after by MIL.. not in a mean way, but he won't be dwelling on it like I was...! It's like what others have said, attachment is a marathon not a sprint. We did wait till nearly 4 months for him to be looked after by someone else, and think it's important to remember as attached as he might seem it really has been such a short period, even at 4 months! But unfortunately, at 4 months adoption leave pay decreases even more.... so we need the money!
Swimming is brilliant -we've just started being able to float ourselves with arm bands. Obviously our LO is incredibly talented and exceptional at everything he does. In fact I'm sure no other baby has ever been so talented.