Do you have a smart meter, or have you been supplying regular readings when requested? Did you have a Direct Debit, that stopped or are there simply no bills on your account?
It isn't necessarily the case that they can't bill over 12 months. From what you've written, so long as you've provided regular readings, I think you will have this protection, although as you haven't advised them within a reasonable time that they've not been taking payments, it might go against you.
When they realise the error, they will probably try to bill you the full amount, so you might need to escalate it to invoke the back-billing protection, if it hopefully applies to you.
I went through a back-billing scenario years ago. They were trying to bill me for £6k in a house I rented. It turned out that some years ago the meter had been changed from one with 4 digits to 5 digits, but they hadn't registered it properly on their system, despite meter readers also visiting in that time. The owner of the property didn't flag it up either and had just been entering the first 4 digits of readings, reducing the bill massively.
They said I owed £6k, as they tried billing me from eg. 2101 to 21019! I used my rights to refuse to pay until they provided a correct bill, which took over 2 years. I had evidence to show I'd done nothing wrong and had provided them with the correct meter readings from the day I moved in, also evidenced by my Check-In Inventory.
I was in regular contact with a manager for 2.5 years trying to fix it, but their billing computer system wouldn't accept the corrections.
In the end, due to back-billing they could only bill me for usage in the last 12 months, and as I'd moved out 6 months earlier, this was 6 months usage from the 2 years I lived there. Their system still couldn't generate a correct bill, so I offered a sum of something like £300 in full and final settlement, which the manager gratefully accepted. He even sent me flowers and a gift hamper as it had been a stressful 2.5 years, with regular (sometimes weekly) calls and threats from their debt collectors as the system incorrectly said I owed them the £6k.
To answer your question, I wouldn't move supplier, although it would force the issue, but I would probably let them know by email so you have a record. It will be within your terms of service to advise them of billing errors, as soon as possible. They are often so slow to respond that it could drag on anyway, so you get to benefit from the back-billing legislation for longer as you will have the email as proof that you advised them as soon as you realised - albeit years later.