Oh Alma [flowers 
You're in a very tough situation right now, and I think you just have to give yourself (and your DC) a break - try and just let things go right now. In the great scheme of things, just recognise that there are certain things you cannot control right now
Your DS is isn't really mature enough to understand why you don't want him to eat all these delicious things - and let's face it, they are delicious - especially if they're being given to him by the other person he must love most in the world.
Is he really fat? Does it really matter if he eats these things for the moment, assuming he has an otherwise healthy diet?
You haven't damaged your relationship - you've just shown him that you are a normal human being, like all the rest of us, and that some times you get angry.
You apologised at the time, but I also think now it's time to sit down with him, when you're both calm - and make sure DH isn't around - and apologise again, but explain why you got so angry and what you're trying to do by limiting the sweet stuff.
I'd also make sure that you don't have stuff like cereals in the house. Not easy if your H is doing any of the shopping, but if you can, then this is where you can exert your control. I would, though, remember that your DC are still young and will be able to tolerate higher levels of carbs than an adult might - and it's difficult for them if they're not able to eat things that they like to eat, even if they're things you'd prefer them not to!
I'd also be trying to have a conversation with your H about what he's doing - does he realise that DS has a problem with over-eating? Does he not love his DS and want to help with this? Or is he just doing this to spite you? Which is a horrible tactic if it's potentially harming his son 
Stepping back from this a little, has your son had any assessment for his eating? I believe that compulsive over-eating can be a psychological problem, and not just one that's in response to his brother's MH, but it can also be a genetic condition.
I've never been in your situation with my DC so can't be of more help, but I can really empathise with how you are feeling. Nothing, nothing has ever made me more angry than my DC and their behaviour
. Dealing with anger in a 'normal' situation is hard enough, and I'm very ashamed of how I dealt with them in some cases, so for you it must be even harder.
So - don't be so hard on yourself. Hopefully you won't be in the same situation for long and your H will have moved out - go with the flow as much as you can until that happens.
Not sure if any of that is helpful, and I fear it's a bit clumsy, but know that you're among friends here, and you can rant and rage about anything without any of us flaying you