It's used against him because at the time the conversation took place, both he, the future King, and Camilla, were both married to other people. He was committing adultery and this was unacceptable especially as he would one day be the head of the Church of England.
With respect - if the purpose was to prove adultery then the conversation itself did not need to be transcribed and published. It was published because it was salacious pure and simple. I have no time for either of them - I still think that publishing that conversation was appalling.