My youngest is 5 and still wakes nightly. Not quite the same as your dilemma in that he just wants a cuddle and reassurance. Now he’s bigger he just walks himself through to our room. When he was still in a cot bed, we had to go lift him.
I have no real advice OP, have you spoken to the doctor about his wakings and how upset he is?
Given how upset and grumpy he sounds on waking, I’d guess he is being woken.m by something - noise, too hot, too cold, thirst, hunger, pain or bad dreams and then gets himself in a state about that and needs to go downstairs to ‘settle’ or reset himself.
If I was you, I’d try trying to predict when he will wake up, you say it’s every other night, and being there beside him before he wakes. Maybe your presence would reassure him everything is ok before he gets upset and you could get him back to sleep.
You could also try moving to a single bed now. Mine went from cot bed to single at around 3. The change may just give him more space to move in his sleep and it ‘might’ stop him waking. Who knows...
Would there be anything about his medical condition that could cause him to wake?
You also mention on nap days, he sleeps better. Any chance of him napping daily? Even if it’s in the car or a quick nap.
Just trying to think of everything for you!