You're not mad!
I'm a single sahm with a 2-yr-old. When I got myself together after leaving my ex-h when lo was 2 months old, I started looking for part time work at the level I left on mat leave. Public sector was the best option, but every post I interviewed for went to internal candidates. So I started looking at positions a bit lower and then I was over-qualified. I went for loads of interviews and got heartily fed up.
And then I did the calculations and discovered it wasn't worth it for the money.
But it's the self-esteem.
So I started my own business! It's great - I work as much as I want, when I want, and organise my events so that I can take lo with me - toddler groups etc - so he gets to play, I get to meet lots of nice new people and earn money without forking out for childcare!
My Mum has lo one day a week now as the business is really starting to take off (and I'm always looking to recruit so let me know if you're interested!) and I really enjoy the day, even though I'm just on the phone and in front of the laptop working.
It'll all be ok in the end, but it is an uphill struggle. And of course things will change again when they go to school - I have friends leaving work when their lo is 4 ish, as they realise that they'll have so little available time when their lo goes to school!