I used to love how rewarding my job was but the reward is dwarfed by the other shit not to mention constant worry
That's exactly how I feel about teaching! The being with the children part is amazing, but totally dwarfed by the rest of it and you know when you're wishing that part away so you can try to make a dent in your massive pile of 'other shit', you know something is badly wrong. Teaching wasn't like this when I started which is what makes me most sad.
It's important though, not to compare stress or try one-upmanship with it, I think. Maybe it's a different stress to the high-stakes stress of operating on a patient who might die, or representing a person who might go to prison if you don't do it well, but it's still stressful. I wouldn't presume to say I would find it more or less stressful than your job because I haven't done your job, but you appear to be suggesting that you'll be ok with it, despite having never done it, because you're used to stress. The fact you left due to burnout is important not to forget.
The stress I feel is the 'boiling frog' analogy. I coped with the increase in paperwork, observations, PMR, increase in parental contact to 24/7, the prescribed scripts of schemes of work and strict timetabling. But that doesn't stop, it expands and the micromanagement gets continually worse, the complaints get worse, the behaviour gets worse, your TA gets removed, you don't just have 4 pupils with SEN in your class, you have 6/7/8/9 and still no extra support, but you are still expecting to make all children make progress, and still expected to stick to the script, and you still have to do displays, photocopying, plan for your unpaid curriculum areas, prepare for Ofsted, parents evenings, report writing, keep on top of marking ready for data drops and book scrutinies, and plan topic days, lead trips, plan concerts and assemblies, oh and another child with an EHCP has joined your class, taking you to 31 with no extra support. It's the relentless creep of workload that I find stressful and am often ill during the holidays because my body feels like it just packs up when I stop the plate-spinning for a few days.
You might be fine, but don't dismiss the stress just because you have different stress in your current job.