I'll start by saying that I have bipolar & PTSD, and neither are in a particularly good place right now. I've also got a baby.
There are a couple of people who are pretty bad at triggering poor mental health episodes, that I can't avoid. I try to keep my distance from them usually, but now that I have a baby, they will literally follow me anywhere.
They hound me for the baby. To be polite, they're always allowed a cuddle, but they'll hound me afterwards to have him back. They'll come and just try to take him off me, or start with the "Oh just give him here", "Let us have a last cuddle" type things.
How do I nicely decline? Sometimes I just need to decline, for either/both of our sakes. I am doing my utmost to avoid them, but it's not possible all the time right now.