I have an olive tree in a patio pot. Neighbour has 4 olive trees in the front garden. Both face southwards. Neighbours olive trees are thriving on low maintenance in gritty soil on a slope and I suspect they do better in the ground than in a pot. You could a) repot into poor sandy gritty soil or b) plant out into well drained gritty soil.
I had a little olive tree in a pot. I planted it in my south east facing garden in heavy clay and it is now about 3 metres high. It has shown no signs of damage in the last few winters when temperatures have been down to -10. It is a lovely tree now and even gets tiny olives on it.
If you are not too far north you could plant it out, they are hardier than people think.