Given that you still talk to and see your father, I think it would be difficult to tell him you are getting married and he isn't invited without putting the relationship you do have in jeopardy.
I notice you say you don't want him there because of your mum and her family, not on your own account. I would hope that your mum and her family would understand that despite what happened in your parents' marriage, he is still your father and should be invited to your wedding.
If/when you have children it can be difficult to compartmentalise your lives. My PIL divorced acrimoniously (at least on one side) several years ago, but I was determined that when it came to events in my daughter's life (like dance shows, etc) they would both be invited and expected to be civil to each other. I was nervous the first time, but we all survived!
I am assuming your parents haven't been in the same room since the split? Maybe you could talk to your mum about your worry that it would be unpleasant for her to have your father there. She might surprise you by her willingness to 'rise above'. My advice would be not to leave it up to her whether he will be invited or not, however, and I also think it is fine to not invite his partner/family.
If you do decide not to invite you father, why not arrange to meet him for a drink or meal with your fiancé and just tell him straight out that you are getting married, but don't want him at the wedding because you think it would be too hard for your mum. I would expect him to be hurt or angry tbh so you will need to be ready to deal with that. In my opinion, it might be less drama just to invite him than end up feeling guilty and stressed in the run up to your wedding.
I hope you can find a way to resolve this that you find acceptable/can live with. Remember that your wedding is you and your partner's day so and you should put yourselves first. Good luck! 