Some babies need to be actively taught that in order to go to sleep he needs to be still. This is a rational extension of logic for you as an adult - you know that if you want to go to sleep then you need to lie still. This isn't a natural or logical step for many children, they don't have that ability to rationalise and its not something that is inherently known, they have to learn.
Being still in order to sleep can often be compounded by the fact that some babies use self-rocking and thrashing as a self-soothing mechanism, if they have no other means to sooth. And that movement, hinders the ability for their muscles to physically relax in the way they need to at bedtime. My firstborn used to thrash her head side to side when over-tired. I realise now this was a self-soothing mechanism but as a FTM I had not realised she needed more help to sooth and comfort to sleep calmly (I embarrassingly though this meant she needed more movement and fuss to go to sleep - she need less fuss and more calm stillness).
So my advice OP - is to help your baby learn to be still and calm. The calm comes from active sucking on the dummy and you not making any unnecessary noise or movement (inc when soothing him). The still bit needs some creative thinking.
In a younger baby, a swaddle serves this exact purpose. Did you use a swaddle consistently when younger? If so, then bringing the swaddle back can help. Just remember to undo the swaddle as baby calms and goes to sleep - since swaddle use isn't safe overnight once rolling. If you've never used a swaddle, it might still be helpful for times when overtired screaming is at maximum - wrapping a child is a great way to calm them down enough to allow other forms of soothing.
A progression from a swaddle to reduce rolling around is a tightly tucked in blanket. To transition out of a dummy I used to use a cot sheet (over a sleeping bag) tucked in so that the long edge went all the way around and under the mattress - its a really snug feeling.
After that, you're looking at ways of using your own cuddling as ways to keep baby still and calm. A sidecar cot allows you to cuddle close to baby (and extract yourself afterwards) and your cuddle can help baby stay still.
You can also use your hands. Having two of babies hands in one of yours stops arm thrashing around. Using your other hand lying firmly on babys legs stops leg thrashing. This isn't a long term thing, it's until baby learns that he needs to be still in order to go to sleep - you need to go through a process of teaching it.
Everything else you do wants to be as minimal as it can be. You go up a ladder of steps according to how distressed baby is.
- If baby is still and actively sucking (so not crying), you just lie on your bed and wait.
- If baby crying, focus on actively sucking the dummy. That's what solves the crying so it doesn't need loads of fuss, just tapping on dummy and waiting, Or remove dummy and get baby to take it in an active way.
- If baby is not crying but is fussing, a firm hand on baby's chest helps reaffirm stillness and reassure of your presence.
- If not crying and making bigger movements, your aim is to help baby be still and quiet. So you find yourself needing to keep babys arms and/or legs still and also making sure baby keeps active sucking. Through this you don't fuss, on the whole you try and stay still and quiet yourself. Make sure the room is dark and has no stimulation - don't get trapped into the idea that you need to be patting, shushing or stroking if baby just needs to be still and suck a dummy.
- If baby wont stop crying and usually takes a dummy, it's probably due to being over tired (unless hungry?). That involves changes you cant make in the moment - for example more daytime naps, less awake time between naps etc. In this case you may need emergency measures that you wouldn't routinely use to get baby to sleep (like wrapping baby mentioned earlier, or picking up and rocking, pushchair etc) with the view that you just get through and then make the necessary changes to routine so baby isn't over tired next time.