I'm the only one of my old friendship group with a baby, almost all the other parents I know are at least 10 years older than me, and to top it all we didn't even make it to DD's first birthday before calling it quits.
I actually (naively) thought that because I am generally an alright person with a decent career and great education behind me, people would not stigmatise me in the same way that they might the 'stereotypical' single mother (not that that is right either, of course). How wrong I was!
When people ask how I am, I usually say 'oh, we're great, DD's dad and I are no longer together, but we find things are better for all of us this way' and that seems to deflect the negativity. But I do get pitied, and I can guarantee that at some stage over lunch/coffee/any social contact, the other person will say (condescendingly) 'I just don't know how you manage', as if life is nigh on impossible for anyone without a husband...
Of course, I'm the slag who got pregnant after 5 weeks with a new partner (I rarely tell people that bit!), never been married and never intend to be either.
OP, I do find that people rarely want to hang out with a single mum and her kid. They are just used to the couple with children scenario and don't know how to behave around someone they might see as having been dreadfully hurt and 'left holding the baby' so to say. If they stop calling you, you could take the reins and call them, inviting them round for tea with you and your children, for example. If they still don't want much to do with you, that's their problem and their loss but at least you will have made the effort.