You also might find it hard to get back on them if they decide to take you off, if you don't meet the needed levels any more.
Mine was about 113/75 on medication, but I had to stop it briefly for surgery and was told to go back to the GP to restart it afterwards. But when they measured it, it was still in the OK range, so they said to just monitor it for a while. It was regularly in the low 130s over low 80s, which is borderline, but because the average was never quite 135/85, they wouldn't prescribe the medication any longer. I'm worried now as it's so much higher than it was, after being well-controlled for years. I know it's good not to be on the tablets any longer in some ways, but I think I'd prefer having blood pressure well into the normal range!
So I tried instead to do everything I could to reduce it myself, though I already ate very little salt or fat, lots of veg never smoked etc. I added in loads of exercise, lost a lot of weight, reduced startch/sugar a lot etc. But none of it has worked and it remains in the mid 130s over mid 80s most of the time. Too low for medication but too high for comfort.
Might be worth discussing that with the doctor - how low/high will they allow, can you reduce the dose without cutting it out completely, will they restart it if it goes up even if it doesn't get as high as it was originally, etc.
I kind of wish I'd not stopped, although one of them (I was on two) I definitely feel better without (the beta blocker), but I don't like the idea of constantly high-ish blood pressure now. If it's the side effects that are causing you to want to stop, maybe there are things they can do to help that instead?