Having worked extensively with both the victims and perpetrators of CSA I think your concerns, and request for help navigating this difficult situation, are valid. This man is a convicted paedophile and whilst he is entitled to live his life and be present in his child's life (assuming there is no Court Order preventing him from doing so) it is not you're job to be 'inclusive' and welcome him into your social circle. One of the consequences of his decisions to act on his sexual feelings towards children, aside from the obvious legal ramifications, is that people may not want to associate with him.
Whilst you are correct that the mother of his child is not responsible for his offending, if she has chosen to stand by him then a natural consequence of that choice is that her social life, and potentially that of her child, is likely to be affected. Sadly, I have been involved in many cases where the wives/partners of convicted paedophiles have been in denial or minimised the seriousness of their offending so she is not necessarily a protective factor.
Rates of re-offending for men who sexually abuse children or view images of CSA are high. If he is genuinely committed to not re-offending then he should be taking all possible steps to limit his contact with children other than his own.
If your child is invited to the other child's party you could attend and make sure you keep a very close eye on your child. Personally, I would find it difficult if I was hosting a party and this man attended with his child as I would feel obliged to watch him like a hawk, in addition to my hosting duties. Attending his own child's birthday party is one thing, but he should not be attending the birthday parties of other children in my view as that is something that could easily be avoided and depending on the terms of his SHPO (Sexual Harm Prevention Order) if he has one, he could find himself in trouble for this.
Personally, in your shoes I would be doing everything possible to limit this person's contact with my children. If this meant not actively encouraging a friendship between my child and his, so be it. It's not his child's fault and it's not fair, but I am not responsible for that child I am only responsible for mine and I would not be exposing my children to unnecessary risk for another child's sake. Statistically, there is someone with a sexual interest in children on every street so you can never eliminate all risk, but it is sensible and proportionate in my view to limit your child's contact with a known paedophile as much as possible and nothing will convince me otherwise.