OP I feel sorry for you.
You say this man is not pressuring you into performing this act, which you clearly have grave reservations about performing. Maybe he is not slamming your face against a wall and telling you to do it but he nevertheless, "keeps coming back to it" as you say.
That is pressure. It is pressure because you have said that you feel he will clear off if you don't do it so the pressure is clearly there.
This is clearly this man's "thing" and so it's obvious he will want you to do it time and again.
You say you have esteem issues and do things to keep people happy.
You do not want to do this and doing it will not make you happy. The only thing it will do is to make this man happy and he will want you to keep on doing it time and time again which will not make you happy.
Look after number 1 and keep yourself happy. You will never be truly happy all the while you are doing things that make you unhappy just to make others happy.
My advice is be strong enough to tell this man that you don't want to do this and get some help with your esteem issues.
If he goes down the road then so what? It'll just prove he didn't care about you.