I cannot dance.
For me, it's not that I don't have rhythm (although I apparently can't spell it, that took me 5 attempts!). I play the drums, I can definitely coordinate my limbs.
For me, its that I am aphantasic, so I can't picture anything, so when I'm dancing I have absolutely no way of picturing what on earth I'm doing. Put me in front of a mirror, and I'm fine, I have a reference for what my body looks like and what it's doing.
So generally, when I'm dancing, I look a lot like your boyfriend @YourChicCat . I'm stiff, and wooden, and look uncomfortable. I'm not uncomfortable, I just look it. I don't find it embarrassing, and it doesn't sound like your boyfriend does either. Why would it embarrass me, everyone is shit at something. I'm good at lots of other things. Yes, people take the piss sometimes, but for me, thats part of the fun of being on the dancefloor at a wedding, laughing at each others piss poor dancing!
I do get what you mean about it being a feature of a previous relationship that you miss in this one. I had an ex girlfriend who I'd sing with. The relationship was a car crash, but we'd do the washing up together and just sing, and it was fun, and brought us closer together. DP and I do not sing together. DP cannot sing, her singing makes my dancing look good.
And yes, every once in a while I look back nostalgically and think of my ex and I singing, while I'm in the kitchen singing to myself. But DP and I do other stuff instead, and we have so much more fun than ex and I ever did.
It sounds like this is a good relationship for you @YourChicCat , why fuck it up for the sake of a single skill he lacks. Make your peace with his crap dancing, and lean into it. Everyone has flaws, they're a part of what make us love each other. His crap dancing is part of what makes him human, a unique individual, the person you fell for.