Yes, you'd be mad
. But I think you can do both...
I have freelanced for 7 years now, and I'm not about to go back to a staff role any time soon. BUT, I did that with the safety net of doing my old staff job on a freelance basis for 6.5 of those 7 years. This was the best of both worlds in some respects - I had regular retainer work, I also had a title I was on the masthead of that I could use to secure contacts and invites (much easier as a staffer than a freelancer often). I then did other, mainly corporate and event, freelance work around it.
It was far from ideal, the negatives being often having too much work, being reluctant to turn down well paying occasional work but still having the low-paying regular work, so often swamped, working lots of nights and weekends etc. But organising childcare around freelancing is almost impossible otherwise. The past 6 months I've been reliant on generating my own work, and although I'm busier than I'd expected I have kept my childcare up which wold have been a financial hit if I'd had a longer quiet period.
I presume you know how much your title pay freelancers? And how many commissions they are likely to individually get per month? Could you live on that?
My suggestion, if it is any use to you at all, would be to increase your GP days to two a week, and use your 4th working day to begin freelancing - if writing for competitor titles while you're still employed might be a problem, then writing about your niche for other outlets, corporate work, copywriting, whatever you fancy/think you could pitch for.
Then as that builds up you might be able to go down to 2 employed days, and swop the GP day when you're in the office for a 2nd freelance day. But, assuming you are only paying for the CM days, you wouldn't be losing money.
Only when you find yourself turning down large quantities of work would I quit your regular job. And even then I'd ask if you could do it on a freelance basis instead!
And don't forget - as a part time worker you still have paid holiday, sick leave, maternity leave, pension... As a freelancer you'll get nada.