How often does your dd have contact with these grandparents? Do they have unsupervised contact with her and does she have overnight stays with them?
Notwithstanding the above questions, in the circumstances such as you've described your dd's paternal grandparents have no legal right to have contact with her.
If they wish to apply for a court order they must first ask the Family Court for leave to commence proceedings and it's by no means certain that permission will be granted.
However, if the Family Court grants them leave to apply for a child arrangements order* they will first be required to undertake a compulsory Mediation Information Assessment Meeting (MIAM) to which you and your dp will be invited to attend and you will be at liberty to give your views as you've expressed them here.
If, after any such meeting, they proceed further it is highly unlikely that any Court would favour them over you and your dp.
Unless they intend to represent themselves in which case I shall put my mind to helping you have them laughed out of court these proceedings do not come cheap, as they will discover if they decide to consult a solicitor who, given the zero limited chance of success, will most probably dissuade them from making any application and steer them towards mediation as distinct from the MIAM mentioned above.
I suspect that, unless you are able to block them entirely from phones/emails etc, you and your dp will be subjected to a load of bluff and bluster interspersed with assorted hollow threats, but please don't worry as they won't succeed in law and, for the record, they cannot apply for parental responsibility as your dd does not live with them and they are not raising her.
*Child arrangements orders have replaced contact orders - those who hold the latter orders do not need to re-apply.