then I end up shouting and he will go to sleep
Maybe he's unwittingly desperate for you to get tough earlier on. When you order him to go to sleep he complies. Could you order him earlier in the evening without the shouting but with a low very firm voice?
Perhaps there has been too much "wiggle room" and the volume of choices and options has been feeding into his worry and compounding the problem.
What you're hoping is for him to say "hey, do you know what, X is a really good idea! Nice one, mum! I'll just go to bed now and see you in the morning. Sweet dreams, all".
It's just not going to happen.
Perhaps if it is black and white he may respond better? Some kids really crave strict boundaries and part of their bad behaviour is trying to teach you that that's what they want and need.
I had a 12 year old on a trip having a full on tantrum at 11pm refusing to go to bed. We were hundreds of miles away from home. Tried everything - chatting, calming down, special breathing techniques, hot drink, cool drink, books.... In the end after about an hour of this I just thought I was not going to be up all night with one child when I had 60 to look after the next day. I got up, ordered him to his feet and told him that everyone was going to bed now and so was he. To my surprise he got up and went to bed and that was it. He was fine. I was annoyed with myself I hadn't done it earlier and saved an hour of hysteria and endless options. I wondered if part of his behaviour was a desire to be treated more firmly - he almost seemed relieved / more relaxed to be snapped out of it.
That's one child and not yours - but perhaps a different approach earlier on?