I feel for you - it's so draining when every bedtime feels like a battle!
My DS is a bit younger (3.8) and is better at bedtime now, although he still has his days, so was even younger when we used to have bedtime antics more like yours. So I'm not sure how useful I can be, but here are some things that worked for us:
To get away from me having to lie on the bed with DS we did it really gradually - trying to go straight to leaving the room didn't work for us. So first I started sitting up straight on the bed next to him - he would ask me to lie down but I just said I was more comfy sitting up. Once he was all happy with that, I moved a chair right next to the bed and would hold his hand. Now he hardly ever needs me to hold his hand and I am trying to leave the room for short periods ( only a minute or so at a time so far!
To get him lying down rather than bouncing around, we got a CD player and took up some of his story CDs (we have tons for long car journeys) and let him listen to one once the light is off - but if he starts messing around the story is turned off - this is quite effective as he loves his stories, so he often fell asleep part way through the CD.
Other tactics were to say I'd only sit with him if he was lying down trying to sleep - going into my room if he didn't listen, coming back if he got back into bed and lay down. Unfortunately this sometimes resulted in tears, which you say you are trying to get away from.
If DH was home, and DS is nowhere near sleep, then sometimes he'll take DS downstairs, but say it is "adult time" so he has to sit quietly on the sofa, no toys. A combination of boredom and wanting mummy generally soon sends him back upstairs!
Also, just making it fairly dull - no conversation, just repeating that it is sleepy time, just water not more milk if he says he's thirsty, etc.
Good luck!