In the UK grandparents have no legal rights unless they've been extraordinarily involved with the child's life, like living with or providing daily care, iyswim. I think that for her to petition for custody for herself she would have to prove that both you and her son are unfit parents. I'd say the likelihood is that she will help him petition.
You can't really stop them from 'trying'. They do have the legal right. Honestly, the best way to be prepared is to seek legal advice right now. You need to find out if denying access now will have a negative effect on custody later. You need to tell the solicitor that you have concerns that the father will not return the child to you if you let him have unsupervised access.
If you have to beg, borrow, or steal try to get an appointment with a solicitor. Call WA and see if they have a list of low/no cost solicitors. I think I've heard someone mention the Gingerbread Trust as an organization for single parents?
But remember that it takes quite a lot for a parent to be declared 'unfit'. Courts and SS routinely find parents 'fit' who have filthy homes, send their children to school in dirty clothes, and feed their children crap. They've even returned children to drug users. You don't do any of that. You have a tidy home, baby is clean and well fed. You take care of yourself both physically and mentally. No one is going to find you 'unfit'.
Please don't defend yourself against his accusations, other than to shrug, roll your eyes, and say 'Whatever' or 'Consider the source'. Don't allow yourself to become angry or upset. You won't convince anyone who is on their side and you won't need to convince anyone already on yours. Just live your life.
I do think, however, that you can probably expect that ex and his mother will probably be making a complaint to SS that you are 'not coping' or that the baby is 'in danger'. Remember that SS has to investigate these things. Just be polite and cooperative if that happens.