We had the same scenario with a baby who would only sleep in a sling, and we would spend our evenings keeping him on us in a sling as we ate dinner in silence! I totally understand where you're coming from.
This is just our experience - I know every baby is different so not guaranteed to work for yours, but I remember the same thing happening to us and it really was a gamechanger with our little boy who is definitely a koala baby. Worth a try, hey?!
On a good friend's advice, at 4 months, we started a proper wind down evening routine. 6pm playing in the nursery, low lights, nothing too exciting, short feed. 6.30pm bath time. No bright lights, just led candles and a nightlight, plus the same lullaby playing on repeat. 6.45pm into sleepsuit and sleeping bag. Then into blacked out bedroom with white noise playing (continuous white noise, not just a 20 min one). Feed to sleep. Once very asleep (i.e. you can lift his arm and it flops down), transfer into Sleepyhead pod in sidecar crib. I found lifting him up and lying him on his side worked every time. If he stirred, I would put my hand on his arm quite firmly and this would settle him.
This gave us our evenings back and has worked ever since. The principle behind it is to be very repetitive and consistent, and every little detail is a sleep cue telling him it's bedtime. As bath is only once a day this is the big cue, so important to get this into place.
The Sleepyhead gave him a sense of closeness, and if you can get something smelling of you next to the Sleepyhead, even better.
At 5 months, when we got into bed, it was waking him up, so now he sleeps on his own in the spare room for the first few hours, and when he wakes up for his night feeds, I will either cosleep with him or re-settle him depending on his mood.
Good luck!