PLEASE DO NOT GO
I was in a very similar situation and posted for advice here. My issue was almost word for word the same as your issue.
I went to the event and it was the worst thing I could ever have done. It was like knowingly flicking the first domino in a long line. I too was all he won't do anything, it's more my anxiety, my job is important, I can be professional.
This is what happened to me.
I went. Of course he was there. I was massively 'triggered' if that was the word by him being there and felt absolutely shit. I compensated by drinking.
He came over to me and was super charming and apologetic. He asked me out for lunch.
I wanted to make it all feel better and thought its only lunch. It will help me feel better.
I went for lunch. He went into pursuit mode. We slept together. He went cold and abusive again.
I fell into a worse downward spiral of feeling terrible about myself.
All the good work I'd done by putting some distance between us was gone and I had to drag my further flattened self esteem back to the already low level it was before I saw him again.
You think you can cope with seeing people like this but until you are totally over it. By which I mean properly in love with someone else and not still vulnerable you need to have absolutely nothing to do with men like this.
Please stay away. Don't judge yourself as being weak or mistaking yourself professionally. It is a decision you take for yourself and because you put yourself first.
I had advice here not to go and I wish I'd listened.