I'm so sorry, you never needed this on top of everything else you've been going through. You must be in shock
More birth parents than you would think do find out where their children live and it's not always a problem - some will understand what a bad idea it is to turn up at their childs house and not want to do it for their childs sake (or would find it too painful even if they wanted to), but even those birth parents who wouldn't think twice about turning up, often live such dysfunctional lives that they don't ever take any action - 'finding' their kids turns out to be all talk, because they can't actually find the will, time, and money in their lives, when they're dealing with their own addictions or MH issues etc.
Of course, sometimes it is serious enough to need action and you might really have a problem. And sometimes even when there isn't as big a security issue, families don't feel comfortable looking over their shoulders all the time. Obviously you've only just found out, and I think having a few more days to think things over is good, and decide what you think should happen next.
You know that social services are responsible for this, so they need to take responsiblity. An apology doesn't really cut it at this point. They might at least be able to tell you whether she is still in prison or not - if they know, they should tell you. What are they going to do about this? At the very least, if they haven't made it very clear to her that it's not appropriate to come near the children and how serious the consequences could be for her, they need to do that. As Happy said, frankly they need to be paying costs if necessary
Because you live on a main road, anything odd is also very likely to be something to do with your neighbours. Don't panic. However if you are worried, and to be honest, if your children's birth mother has a history of violence I would consider this anyway, you can contact the Police yourself and ask to meet them and talk to them about the risk to your children. I know a family who talked to the Police following this kind of a slip up (where it was felt there might actually be a kidnap risk) and the Police were definitely helpful (and understanding) and told them what to do if xyz happened and they would know it was a priority for them to deal with etc. They actually moved house in the end though.
In your situation, I would also go into school and tell whoever needs to know about this risk, school should to be more vigilant at pick up time etc.