OP if you can still feed, clothe and shelter you and your children with the part-time job I think you are right to make the leap. If, as a parent, you find it difficult to give a healthy amount of weight to your own well-being, think about these points:
1 The simple fact that the part-time job has better long-term prospects makes it worthwhile.
2 Sacrificing material extras in order to achieve happiness, and commitment to a passion are valuable choices for your children to see you making. They will internalise a different lesson if they see you prioritise material extras over personal happiness.
3 It's not always popular to say on here but if all basic needs are met, then your children will be happier if you are happier in yourself.
Speaking personally, I have a family member who is a published author and refers to his craft as 'typing practise'. Watching him get up every morning at 5am to sit at the page has had an immeasurable influence on my own attitude to life and work.
Finally, please do not listen to people like Sally, who is probably just bitter about giving up on their own dreams out of misplaced fear or obligation.
Good luck! Let us know if you become the next JKR!