As people have said on here - his argument (that people can't choose what they find attractive) isn't all wrong.
To take up his argument:
It is utterly wrong to compare padeophilia and homosexuality; this is an old homophobic view -seeing attraction to the same sex as a perversion and ignoring the whole matter of consent.
Paedophilia is a perversion. It is comparable to sexual-attraction to animals (since they also cannot ever consent).
It can never be accepted as natural and dealing with it is the responsibility of the individual who feels the attraction - just as attraction to someone else's diamonds/fast-cars doesn't excuse the theft of these things. Paedophiles who act are abusers not victims of their natures.
On your reaction to him:
You listened to his argument - as a sympathetic human being would - but clearly felt uncomfortable about something. (Luckily, some part of your mind kept working on it after you left the subject.)
You were at first reacting to what he said but then later to what he meant.
You were right.
It isn't the argument that is most worrying, it is his introduction of it to you - a single mother with a history of abuse- in this 'random' way.
The fact that he says he does this to others shows that it is not random; it is deliberate. This makes it much less innocent and likely not merely an intellectual matter to him.
Your instincts on this are sound.
Well done on getting back your balance through all your hard work with the freedom prog - and on protecting your children so effectively.