I walked away from a well paid, high profile job I had spent years training for when my first child was born.
I felt exactly like you - completely conflicted and to be honest, terrified at the prospect of juggling a job I knew to be incredibly demanding on a young, single person wth my new life and priorities.
I won't lie to you - I found it difficult to walk away from...and I didn't just walk into some sort of idlyllic 'happy ever after' situation as a SAHM, either. It was hard losing my salary (although we have managed admirably, actually, with some lifestyle adjustments) and it was hard losing my status in social situations. I realised that 'what people thought' mattered a lot to me - too much, probably - and that was a big thing for me to get over.
However, in the long term, it has been a fantastic move for me. I have (eventually) gained a better perspective on my life and got some balance. I had a second child after leaving my 'great' job, which has been wonderful. I love my little family, and have really cherished the time with them at home.
Amazingly - which I never envisaged happening back when I walked out on my job 4 years ago - I am now really ready to go back to the world of work, and have got another great job. Very lucky, I know, but I think it proves that staying at home for a while with your babies doesn't mean the end of your career. Far from it. I feel like I am a more mature, better human being, actually, with a better perspective on things. I gained lots from going with my instincts. I'd say - if you can honestly make ends meet without your salary - that going with your gut might work well for you, too.