Thought we were past this - nearly 3yo waking every night and sleeping in our bed - this can't go on but how?

[sleepychunky]

DS2 will be 3 at the end of the month. He's slept in his own room since he was a few months old. We've had no problem with him sleeping for absolutely ages (like a couple of years) until recently. He was in a cotbed until a few weeks ago, but suddenly started making a fuss at bedtime and in the night if he woke up - literally screaming that he didn't want to be in there and he wanted to sleep in our bed (bit of background which is probably useful - when DH looks after him during the day, which he does quite a lot because he does shift work so is often at home Mon-Fri daytime when I'm at work, DH used to give him a nap in DS1's bed or sometimes in our bed).
I dealt with those night time wakings with a variation of CC (don't shoot me, this isn't about CC) - just going in and telling him it was night time and to lie down, ssh etc. and then returning every 3 or 4 minutes until he was asleep again. I can do that - I can deal with the screaming as I know he's not hurt or needing a nappy change or anything like that. Also at this point he was still in the cot bed so couldn't get out.
DH however really can't handle it when DS2 is crying and gets incredibly worked up about it. As a result there were a few times in the cotbed era when DH1 was in charge and got DS2 out to sleep with him in our bed. We thought that the reason DS was making a fuss might be that he didn't want to be in a cotbed when he knew that everybody else (us and DS1) had proper beds, so last week we turned the cotbed into a normal toddler bed so that we all had the same kind of bed. But every night since then DS has woken up in the night and screamed and screamed to come in with us. DH has been around all week and has either gone into DS's room to lie down on the floor next to him until he's gone to sleep again, or has brought him into our bed.
DH is on nights now so isn't around, and it happened again last night. I ended up bringing him into bed with me because he was screaming so loudly I didn't want him to wake up DS1, but I don't want this to happen every night. I am a very light sleeper and sleep incredibly badly if I have a wriggling small child with me, plus he's nearly 3 and is very capable of understanding rules. I just don't know what to do to get him to sleep in his own bed all night without any approach causing major problems for everybody else. My bed is a sacred space to me as I am always so shattered by the time I get there, and I really want to keep it child-free!
So, what approach would you take to get DS2 to stay in his own bed all night? It's so long since we had any problems with sleeping I really don't know what to do. Thanks for reading - sorry it's a bit of an epic!

[Meggymoodle]

Is he too young for having incentives? I can't remember but I think it's about that time we instigated them for DS. At the moment, he has got to wait until his groclock is yellow before coming into us. If he does that for 10 days he gets to watch the lion king. (We don't have much telly in this house so that's a major treat).

Might be worth doing 3 nights at a time and a prize (trip somewhere or a small toy or something).

[Pancakeflipper]

Are they cold? My DS2 used to come into our bed a lot then I realised whilst we were all snuggly toasty he had a room that got really cold.

[sleepychunky]

Thanks both. He's definitely not cold as his room is the smallest, double glazed and always really warm. I was thinking about incentives and doing a bed chart where he gets a sticker for every night he stays in bed all night and once he has 5 (or 3 or whatever) in a row then he gets a new toy. I might make a chart before bedtime tonight and explain it to him and see if it makes any difference.