Your manager is right:
As PP have explained, these are not hours you were asked to work (for which you should, naturally, be offered overtime pau or TOIL) but hours you chose to work on your own initiative.
Thought I'd give you some of the reasoning why it doesn't work like this, though; maybe it helps someone understand what the issue is:
In a nutshell, what allowing this would amount to is a right to determine your own hours. Which, to be fair, some jobs and employers allow. But it's not the default contract.
Businesses will often have a need for employees to be available at certain times (e.g. opening hours), so they have a reasonable expectation to determine that these are the hours they pay their staff for.
More importantly, perhaps, is that - depending on your contract - hours eligible for TOIL may also be hours eligible for overtime pay. There may be neither a business need nor budget for these additional hours.
Your manager is wrong in one respect, though: IMO, if you've been doing this for a while as you say, you shouldn't have gone unchallenged. If I see an employee of mine work overtime I've not authorised, I will ask them why. If there's a valid business need, I will either find a solution that means they won't have to work overtime or I will make sure the hours are accounted and they're being compensated but will remind them that I expect to be notified if this becomes necessary (for their sake as well as because I need to know if one of my projects is in danger, and excessive overtime is a definite tell). If there isn't a business need, I don't allow them at work. That's because as their manager I'm meant to ensure to the best of my ability that my employees stay healthy and don't burn out and that my company isn't in breach of any laws.
So, YABU to want the time back - but your manager shouldn't have let you build it up in the first place.