I didn't return to work after maternity leave and have never regretted it. I had had enough of my job, it involved long hours and lots if stress and emotional energy, plus we wanted to move house to be closer to family anyway, which would have been an insane commute if I'd gone back. We were fortunate enough to be able to afford it, and crucially, I was very much what I wanted for myself and dd, so it was a no brainer really.
It isn't necessarily the easiest path to take - it can be lonely, especially if you have a support network of other mums from your antenatal group, which then dries up when they all go back to work. As you're already finding you do get lots of comments and judgements to deal with, mostly from other working mums, who may struggle to understand your decision, or are perhaps envious that you are in a position to make it. On good days, you'll just brush these off, but on tough days, you may find they really get to you and you'll have to work hard to remind yourself that you don't need to feel guilty for doing what you believe is right for your family.
I think the thing i found the hardest was the lack of financial independence. Have a proper discussion with dh now about how it's going to work once your income ceases. Will dh's income just go into a joint account which is used for all expenses, or will a certain amount each month go to you to be used for set things? What happens if you want to treat yourself to a beauty treatment or night out with the girls, for example, or buy dh a Christmas or birthday present? Is he happy for you to use his earnings for these things or would he expect you to dip into personal savings - in which case is this something you're willing and able to do? We probably didn't give it enough thought beforehand and it has certainly led to one or two heated discussions and upset over the past few years.
Finally I guess you need to consider how easy it would be to get back into your career ( should you want to). I'm fortunate in that I had a job that is in high demand and I'm unlikely to ever struggle to get back into. It also lends itself well to freelance work, which I've been doing since dd turned 2. At that point it felt right to be doing something other than just "being mummy" and it has been good to have an income again, albeit a small one. When she starts school next year I'll make a decision as to whether I try and find a permanent position or continue with the freelance, but for now things are working well.
I guess for us the decision has also been influenced by the fact we only have the one dc and don't plan on having more, do that's also something to factor in, as it could be the difference between maybe a 2 or 3 year career break or 5 plus years, and obviously also impacts on cost of living.
Anyway, good luck whatever you decide. Just remember it is your and dh's decision, no one else's.