Glad you held firm about not wanting physical contact Nini. It's important that you stand up for only doing what you want to do.
I'm feeling a little [grrrrrr] now. Normally on a Tuesday I take the boys up to the house and NSDH has a couple of hours with them, makes them dinner (well for DS1 while I feed DS2). It always goes quickly, we have to leave when it's nearly DS2's bedtime. I agreed last week that he could come to my flat and see them there, it'll be more relaxed for him hopefully, and he'd get to spend a bit more time, put DS2 to bed, etc. He asked if we could make that another sleepover night, I said no, I felt it was too soon (and still too shakey) to be adding even more to what we're doing.
Fast forward one week, we've had a good weekend. I ask if he's coming to the flat on his way home from work, as we discussed. He says no, what he wants to happen is for us to come to the house, and stay over there. I'm starting to get that
feeling that he's even asking this but I try as positively and gently as I can to say, let's focus on what we're doing for now. Apart from anything, I've only just unpacked the stuff from staying over there on Friday and Sunday night, can't face packing everything again - taking monitors, baby bottles, baby milk, baby sleeping bag & sleeping toys, plus clothes, etc. (Sounds petty, but as well as the effort of having to remember everything I have to lump all this stuff, plus the actual baby, down three flights of stairs to my car, and then carry it all back up again when we come home.)
So we've now had a back-and-forth on the text, him basically saying we should be building on things and moving forward (ie I should be staying over with him or at the very least him staying over with me). And because I've continued to say no, that's not what I want to happen, I'm being awkward: Whatever you want Ponygirl, I can't talk to you when you get this way.
I'm feeling pretty frustrated. He suggested we stay Friday night instead of Saturday because of his sister's wedding - I agreed. He suggested we stay Sunday as well because things were going well and it would be a nice end to the week. I agreed, although with mixed feelings since it was a school night, and it was a bit to-and-fro for the kids. Then he suggests this, which I'd already told him I didn't want to do, and because I haven't agreed I'm causing problems. Now don't actually want him to come at all this evening. But he'll come, and no doubt tell me that I'm being negative and holding things back between us. 
It's not me is it? Surely there's nothing wrong with wanting to wait until things are more stable and consistently good between us before adding another element of spending time together? It's been one good weekend, that's all - great, but we need more good times to follow. Surely part of the effort he should be putting in is to say, ok, this isn't what I want to happen but it's obviously what ponygirl wants so I'll go with it, I want to make her happy. (Rather than constantly push-pushing for more-more-more and making me out to be deficient in some way for not jumping to it.) It's almost as if because he didn't manage to get his own way with persuading me to go to the wedding with him, he needs to assert himself now in another way, by saying 'well, that didn't happen, but instead this will happen'.
Looooooooong, sorry, ranting now....