Firstly remember it is your wedding, therefore you invite who ever you want to make your day enjoyable. We didn't invite DH biological father to our wedding (hasn't had much contact for 20yrs) but we invited his aunt and cousins, knowing his father would be aware but also hoping he knew he would have no place there.
Given that you still talk to and see your father, I think it would be difficult to tell him you are getting married and he isn't invited without putting the relationship you do have in jeopardy.
I notice you say you don't want him there because of your mum and her family, not on your own account. I would hope that your mum and her family would understand that despite what happened in your parents' marriage, he is still your father and should be invited to your wedding.
If/when you have children it can be difficult to compartmentalise your lives. My PIL divorced acrimoniously (at least on one side) several years ago, but I was determined that when it came to events in my daughter's life (like dance shows, etc) they would both be invited and expected to be civil to each other. I was nervous the first time, but we all survived!
I am assuming your parents haven't been in the same room since the split? Maybe you could talk to your mum about your worry that it would be unpleasant for her to have your father there. She might surprise you by her willingness to 'rise above'. My advice would be not to leave it up to her whether he will be invited or not, however, and I also think it is fine to not invite his partner/family.
If you do decide not to invite you father, why not arrange to meet him for a drink or meal with your fiancé and just tell him straight out that you are getting married, but don't want him at the wedding because you think it would be too hard for your mum. I would expect him to be hurt or angry tbh so you will need to be ready to deal with that. In my opinion, it might be less drama just to invite him than end up feeling guilty and stressed in the run up to your wedding.
I hope you can find a way to resolve this that you find acceptable/can live with. Remember that your wedding is you and your partner's day so and you should put yourselves first. Good luck!
My parents got divorced very messily over a decade ago, my dad behaved appallingly and I won't go into detail but the whole situation and the fall out was entirely his fault. My mum is wonderful I'm externally close to her whole family they are all brilliant. I don't really see my dads family maybe once a year. I speak to my dad maybe once a week and see hi Every couple of months. The relationship is an odd one and I have to put a lot of history out of my head and ignore the huge elephant in the room, but we muddle along. I don't want him or any of his family at my wedding it would make the day unpleasant for my mum and the wonderful family that I have grown up with. Now how do I tell him I'm getting married? It's looming rapidly, next month, and he's still none the wiser!? Any ideas? Thanks