My DC are challenging so I have a few things I've learnt over the years.
Agree the Siblings Without Rivalry book is great. Fingers crossed it has some helpful stuff for you.
I would also definitely consider some kind of parent coaching/behavioural coaching if you can find someone nearby before you consider separation - it has to be cheaper.
One really important principle is that if the punishment isn't working, don't escalate the punishment/threat (I totally understand why you have done this; it feels logical) but look at where they are now vs where you want them to be (behaviour wise) and reduce the size of the step, then focus really intently on achieving that in between step. If it's still not working, make the step smaller.
This might be counterintuitive, because for example if they are currently physical with each other and you just want them to coexist peacefully, you might need to temporarily reward verbal aggression, as long as it does not escalate to physical. Or you mentioned name calling so perhaps a better example for you would be to reward if they manage to verbally spar without using name calling. This will sound really weird but if you look at positive dog training, it explains this all so much better than any parenting resources ever do. This technique in behaviourism, where you move closer to the wanted behaviour (in dogs and humans) by targeting something which is close to the unwanted behaviour is called shaping. So you can look up how to do shaping behaviourism. It will probably bring up more dog training results than parenting, but the principles are exactly the same.
For human children, rather than the instant reinforcement of a food type treat, what you can do is use a sort of points/token system where they have the ability to gain a certain number of points per day, or per period of the day (e.g. before school / after school, between meals for holidays/weekends) and points are lost for certain behaviours, but if they successfully do not do that behaviour, they can get the points. Then after the end of a week or so the amount of points remaining determines the level of some kind of reward. These could be individual pots or joint, depending on whether individual creates too much attempt to goad each other into losing points or joint creates too much fighting because they feel the other is sabotaging them. If you really want to evoke a sense of teamwork, have the adults also work on something which they can lose points for. One benefit of a points type system is that they very quickly have another chance to gain more points, but the other is that it is essentially separating it from you, which can help a lot. However in order to keep it separate, you do have to use it almost like a robot would - like a video game score. So establish what behaviours lose points in advance, and stick to that even if they end up finding what seems like a loophole or seems just as bad (e.g. they stop calling each other names but they say things like "I wish you would just XYZ" instead).
The other thing which is important is teaching them a replacement behaviour for what they're currently doing. Because trying to suppress a behaviour is REALLY REALLY HARD. Especially when it has an emotional root and/or is occurring when the person is in an elevated emotional state, and especially when the behaviour has been repeated umpteen times and received a response - imagine drawing a line in pencil and going over it many many many times and pressing hard. Eventually it gets really difficult to rub it out. But painting over it with tippex will work. It's similar for behaviour. Anything which happens repeatedly lays a trail in the brain which reinforces it and makes that pathway stronger. Anything which receives a response of attention, whether it is good (laughter) or bad (being told off) is reinforced, which means their brains regard it as an important, useful behaviour. But brains are plastic and they can learn new rules, it's just that you have to get to a point where the new behaviour is happening more frequently than the old behaviour in order for the older behaviour to fade. You've talked a lot about how their reactions are almost automatic - this is exactly right. They have fallen into habits where they automatically react, and you can't undo automatic habits with reasoning, empathy or talking. If you've ever driven a car abroad, you'll be familiar with how you try to get into the wrong side, smash your hand into the door when trying to change gear, etc. Automatic behaviours take time to overwrite. But if you moved to a country with left hand drive, after a while you'll stop doing these things.
DH and I were reflecting yesterday how we get this arguing and fighting to be seen by them as unacceptable as other behaviours - such as swearing etc. DS1 knows all the swear words and he has friends who swear a lot at school, but he knows it’s not ok. That has sunk in over the years, one way or another.
I think this is really interesting and a useful observation. My guess is that it's partially modelling. I would imagine that you and DH are not rude towards people outside the family and do not swear, but you have mentioned arguing back with DC. And partially that on some level you're accepting the fighting and rudeness. Not intentionally, but there is likely some different way that you respond to swearing that you don't respond to fighting/rudeness, or vice versa. Like perhaps swearing doesn't particularly get you escalated, whereas rudeness does. Or you immediately and firmly react to swearing whereas arguing/rudeness you start to try to solve the problem (engage with the communication) before you hit a point of wanting to shut it down.
IME, stopping arguing back was really gamechanging for myself with DC. Other posters have referred to this as "starving the oxygen" around those interactions and that is exactly right. I don't especially like Supernanny, but if you look up videos of this on youtube, she displays this really effectively - the parents often are engaging with their children in the argument phase and she will step in and say stop, come down to their level, tell them what to stop doing and then take them off to whatever time out spot she is doing in that programme. The best examples are usually in the first week after she stops observing and starts directing, and then the week where she leaves them to it and it all falls apart so she comes and dissects the video footage. The other place you can see what not engaging in the argument/starving it of attention looks like is any modern UK police type documentaries, especially the ones where a suspect is already in custody (Custody Hotel was a good one) although ignore the parts where someone ends up getting wrestled to the floor! I know both of these are quite extreme examples, but they are helpful IMO because they are pretty extreme. When we're faced with a person who is shouting at us and arguing, the automatic response is to revert into a more immature, escalated emotional state ourselves and so we will often automatically respond back in kind. But if you can rise above this and ignore it, it helps a lot.
What it tends to look like IRL is that DC starts shouting/complaining at me about, let's say, they have lost their shoes. Old me would have tried to start off firm and say something like "Why don't you look in the place you were yesterday?" they kick off because it's not really about the shoes, it's about how they have become escalated and have successfully latched on to me as an outlet. So they escalate, I escalate and start using an irritated, raised tone or even shouting back, about just wear your other shoes then, or if you put them away properly, they wouldn't be lost, or come on, just put on ANY shoes, we are late.
My oxygen-smothering response now is to take a deep breath, recognise the problem is emotional escalation, not lost shoes, and not engage in the lost shoe discussion - so I would make my voice deliberately low, slow, quiet-ish and say something like "Please don't shout at me." DO NOT ANSWER ANY QUESTION ABOUT THE SHOES. I find this really difficult to do, especially if I know the answer, and especially if it's a stressful situation like we're going to be late. It does get easier with practice (and it is worth being late a few times, also leaving more time when you know there is potential for explosion is helpful). The thing is that if DC calms down, they can almost always find the shoes or solve the problem for themselves fairly quickly, and/or they are receptive to my help. But if I engage in the emotionally-charged discussion, they remain emotionally escalated and they can't solve a problem like that. So it was a big gamechanger for me to reframe that whatever the problem was originally, unless there is some actual emergency like a bone is sticking out, the problem is NOT whatever they are kicking off about, the problem is that they are kicking off, and they need to emotionally downregulate first. Very little children might be dependent on you to co-regulate, but at 8 and 11, yours are not, and I tend to find TBH that trying to get them to see that the problem is emotional escalation does not help, it just escalates the situation further, so the best thing I can do is just to step away and refuse to engage in anything relating to solving the problem, and just repeat a broken record type thing of "I can help when you speak calmly" or "Stop shouting at me." Sometimes I go into another room and close/lock the door. Sometimes I just very firmly and obviously turn my attention to something else e.g. other DC, emptying dishwasher, packing car.
Now, yes, at first this is going to provoke WW3. This is called an extinction burst. You're no longer giving them the response they expect and they are going to push harder and harder expecting you to cave and give in because that has worked before. It's not necessarily a conscious choice by them - they may not know what else to do. But what they are experiencing is a feeling, and feelings pass on their own. Expecting this in advance makes it easier to hold firm. 99% of things are not an emergency - nothing terrible will happen if they can't find their shoes for a few minutes. My only suggestion for this and the worry about what the neighbours will think is to time it. I bet you it's all over in 20 minutes. And I bet you have about 3-5 times of 20 minute explosions before they start to understand it isn't going to work any more and you do actually find that they try asking calmly.