I'm Equality and Diversity Officer for my employer (a large NGO) so hopefully can shed a little light here on where you'd stand.
Firstly, it's not illegal to refuse to use someone's pronouns. There is no such crime in itself. However, if your refusal to use someone's pronouns can be shown to be part of a general pattern of hostile and/or discriminatory behaviour towards this individual because of their trans status, or towards trans people in general, then that is illegal. And to be honest, a deliberate refusal to use someone's pronouns does unavoidably come across as rude, as potentially hostile, as designed to make someone feel uncomfortable, or to draw attention to the fact they are trans, and it would be very difficult for you to argue otherwise. So while it's not illegal in itself, if things did escalate to a complaint, it could certainly be used as evidence that you have a generally negative attitude towards trans people. Needless to say, that wouldn't look great for your case.
It's worth remembering that different legislation applies depending on who this person is and what your relationship with them is. If you are an employer or a service provider to this person, then the Equality Act 2010 applies, under which gender reassignment is a protected characteristic. Again, the EA doesn't specifically say anything about pronouns, but again, refusal to respect pronouns could be used as evidence against you as part of a wider case. It would be difficult to argue that you are not discriminatory towards trans people if you persistently refuse to respect their identities, for example.
I'm not sure why you referenced the EA guidance on single sex spaces - that doesn't seem relevant here, unless there is an issue of space usage at play? Remember that even if you have reasonable grounds to exclude a trans person from a single sex space, you still can't discriminate against them for being trans - and in fact your refusal to respect their identity could be taken as evidence your exclusion is based purely on discrimination and therefore not reasonable under the EA. If a service provider excludes trans people from a single sex provision, the onus is on them to prove this is reasonable should they be challenged. Again, evidence of previous prejudice against trans people just for being trans would count against you here.
If this person is someone you work with, even if your behaviour does not break criminal law, you might well find you are in breach of your employer's policies on equality and diversity and/or bullying and harassment. Your employer therefore may have grounds to take disciplinary action against you. In an extreme case, they could even fire you, and there is legal precedent for this in the Maya Forstater case where the judge upheld her employer's decision not to renew her contract because of her anti-trans views. The wording of this judgement is worth paying attention to - critically, it wasn't Forstater's beliefs that made it reasonable to fire her, but rather it was the way she expressed them. In short, the judge said she was perfectly entitled to her beliefs about sex and gender, but her beliefs did not make it okay for her to be directly rude or hostile towards trans colleagues. (And again, persistently and deliberately using pronouns you know someone dislikes does unavoidably come across as rude and hostile, and it would be very difficult to argue convincingly otherwise)
So given all that, the real question is, is there actually a reason why you can't just use this person's preferred pronouns? Is there a particular reason you want to be rude to this person, or to make them feel uncomfortable, or to keep drawing attention to the fact they are trans? Do you really think your personal beliefs about sex and gender are always more important than anyone else's right to a little bit of politeness, privacy and dignity? If this person is just trying to live their life and get on with stuff and you're going around deliberately making things uncomfortable or difficult for them, even if that's not actually a crime, it's still pretty rubbish behaviour - and, should there ever be a formal dispute between you, it's pretty clear evidence of your general bad attitude towards trans people, which is not going to do your case any favours at all.