As you say this will have to be very gradual. Has anything been done through the court yet? Or is it a personal arrangement?
I would take it very slow, who will supervise? If you can bear it you will be able to keep note of your sons reactions to visit, and it can be seen as facilitating the contact, and you can have a little more control.
Or is a contact centre possible?
My ds ended up very unhappy having to go through a similar but not identical experience, and like you I felt so torn. It was his dad, but my strongest instinct was to protect him.
As for the other things you mention, there is a lot of speculation, you know there is little point worrying about how he will react. Yes, he will meet lots of different people and be exposed to some things you would not choose for him. But some of it you have to bite the bullet on, if you are going along with contact, which it seems you are, they are some of the realities of it.
I remember I specifically stated I agreed on a trial basis, to be reviewed etc.
Do you have a solicitor you feel comfortable with ?