Being handed a certificate for work you never approved, on your own property, is a pretty unsettling way to find out - and being "in two minds" makes complete sense given the upgrade itself isn't the problem, the process is.
You were clear and reasonable from the start - you agreed to an appointment to explore options, but explicitly said you wanted to see plans and recommendations before anything was installed. That's not an unreasonable ask, that's just basic due diligence for a significant, permanent change to your property. The fact that step got skipped entirely isn't a misunderstanding, it's everyone involved - tenant, letting agent, and installer - deciding your explicit condition didn't need to be met.
A few angles worth thinking through:
This is worth pursuing on principle, regardless of whether you'd have said yes anyway. You may well have approved it with full information, but that's exactly the point - you didn't get the chance to make an informed decision about work on your own property. Letting it go because the outcome happens to be fine sets a precedent that your consent is a formality that can be skipped if the work is likely to be "beneficial enough.
Under a standard UK Assured Shorthold Tenancy (or the new periodic tenancies under the Renters' Rights Act), tenants virtually always need landlord consent for alterations to the property, and a heat pump installation is about as significant an alteration as it gets - it involves external units, pipework, often changes to the heating system, and can affect things like your EPC, insurance, and any future sale or remortgage. It's worth checking your tenancy agreement's specific alterations clause, but the general legal position in the UK strongly favours you here.
The letting agent is probably your first port of call. They're the ones who sent you the certificate after the fact, which suggests they were involved in green-lighting the work or at least aware of it going ahead. Worth asking them directly, in writing: who authorised this, why weren't you consulted per your explicit instruction, and what their process is for ensuring landlord sign-off before installations happen going forward. If they're a member of a redress scheme (Property Redress Scheme or The Property Ombudsman), that's a route worth having in your back pocket if their answer isn't satisfactory.
The installation company may also bear some responsibility, depending on what they were told. Reputable MCS-certified installers (heat pump grant schemes like the Boiler Upgrade Scheme require MCS certification) usually need written landlord consent before installing on a rented property specifically to avoid this scenario, so it's worth asking them directly what consent they were shown or told existed.
Whether it's a breach of contract depends on your tenancy agreement's exact wording on alterations and improvements, but most standard agreements do require landlord consent for anything of this nature. Even if the outcome is something you're happy to keep, establishing this was a breach is useful for setting the boundary clearly, so it isn't tested again with something you wouldn't have approved of.
Ultimately, you can decide to keep the heat pump and still make clear the process was unacceptable. Those aren't mutually exclusive - accepting the upgrade doesn't mean accepting that your consent didn't matter.