I think the thing is that you just have to kind of decide that you aren't going to let things get to you and you have to work hard to stick to that. It is possible.
A good basis is to make your own life and much how you want it to be as possible. When you have the choice, choose only to surround yourself with positive, upbeat, supportive people. Make your environment exactly as you wish, treat yourself well in all respects - doing good things for yourself, eating well, getting rest, treating yourself to experiences and things that make you feel good whether that's a trip somewhere that interests you or sitting in the garden with a good book.
Make the future an open book. Start a course learning something new, make a plan for retirement, think of ways to retire earlier (downsizing/saving money is other ways). Make your now and future so good that the people who get you down are irrelevant.
Also you can limit how things negatively impact on you. If something bad happens allow yourself a short time to think about it - say 10 minutes. Think 'does it really matter?', if not, let it go. If it does matter then make a plan of how to deal with it. Then....when the 10 minutes are up, make a coffee or go for a walk or a swim.......move on. Only allow set times to worry and if your mind strays to negative stuff at other times..do not let it! Get up, get on, be you and enjoy your life.
If you don't deal with this you will take it with you into retirement. There will always be people who will be a thorn in your side wherever you are and whatever you're doing. How you deal with it is the way to peace of mind.
I think we just get to a stage where we want out of work (I definitely did) and when you get there there is no going back. I reckon you're there.