I was in the same boat as you and felt that I needed to meet people who I found interesting. I was meeting people at Toddler Groups and, like you, having a bit of chit chat. I knew that I needed to bit the bullet and invite people round or arrange to meet, but you feel like you are asking them on a date. One other thing was that I didn't really want to, as they didn't seem that interesting. That sounds awful, but after a year of weekly chat I knew that they were different to me, nice, kind people, just different.
I then made a huge effort to meet more lilke minded people and one of the ways I did this was throught the MN local. We now have a fab group of people that meet up quite regularly. We have been out without kids and that really helps. We chat in the MN local section a bit, so get to know each other a little more without children about as such.
I also had a couple of what i call 'blind Mum dates', where a couple of friends had said that they knew someone who lived near me and we arranged to meet up.
So, now I have a few interesting Mum friends in my life and it is so much better. The fact that DD is now a bit bigger also helps, as at first I couldn't string sentences together due to the fatigue and then had to chance DS around too.
You do need adult conversation when you are at home. Its not just about getting out of the house, but also getting some stimulation for yourself.