As Urchin said you can put the past behind you without having to meet him!
If you go ahead with this meeting, make sure it is in an neutral place. A cafe or something. Keep it to a short cup of coffee or something and arrange it for a time when you have childcare so you don't have to take them there.
Keep neutral when you talk to him, keep calm. In your shoes I would find it extremely hard to trust him becuase he is a manipulator.
Write down your feelings ahead of time - anger, hurt, disapproval, maybe even love - and put them in your handbag. Write down specific incidents. And at the bottom, write down Calm, Careful, Neutral. If you find yourself getting charmed, go to the toilet and take out that list, re-read it and take a few breathes until you can go back to being calm, careful and neutral.
At worst, you can cut the coffee short if you feel that you can't handle it any more.
And yes, change your counsellor.
Last thing .. I don't know what forgiveness is very well. But what I do know is that if you forgive someone, that does NOT mean you have to walk back into a damaging situation. Not that you've talked of forgiveness here, but not walking back into a damaging situation still applies.