Yes, I do allow mine to visit friends houses (when its age appropriate, I'd say from age 6-7 maybe) if I know and trust the parents.
Unfortunately, the reality is that abuse can happen anywhere and with anyone so unless you never let them leave the house until they are 18, there is a level of risk that you have to accept, if you want your kids to have happy, normal childhood experiences.
I say this as someone who was abused as a child (similar situation to what you've described actually). I've had two close family members convicted of CSA. Based on this experience, it sort of feels like I should never let my children have contact with anyone but dh and I. But that's no way to live. I'm conscious that abusers could be literally lurking anywhere (they certainly were, and not as far away as at friends houses). I never for a second would have guessed it would happen so close to home, though I did always have an uneasy feeling about each of these men. Thankfully, I trusted that instinct and I didn't let them have unsupervised contact with my dc, and obviously, after I was told about their offences, we never had anything to do with either of them again. I have no reason to believe my dc were ever harmed, but in theory, they could have been - at Christmas dinner, at family birthdays, visits for Sunday lunch. It would have been mad to not let them see family on the risk that a family member could turn out to be a paedo.
There is risk everywhere, and to keep it as close to zero as possible means not living a happy and free life. So you have to find a healthy balance. My experiences have led me to trust my gut and I listen to my gut about people. I do know and trust the parents of dc's close friends. When I haven't had a good feeling about a particular family or situation, no, I didn't allow it.
My eldest had a friend when she was maybe 7/8 who lived with her grandparents, her mum had passed away, bless her, and her dad was 'living away for work' (I expect this may have been a kind way of saying in prison). The grandparents seemed quite overwhelmed, but also quite inappropriately permissive. This 7/8 year old wore make up, had an iphone, a tik tok account, used to walk from their house into the village (about 2-3 miles) on her own, etc. She was very 'grown up' for a 7/8 year old. I didn't know the grandparents well, but there was something just off (besides the permissiveness). I didn't feel comfortable with their situation, and I wouldn't allow my dd to have a playdate or sleepover there. I invited the friend to ours a few times to gauge what she was like and then intentionally let the friendship fizzle out. My gut said something wasn't right there and it was more important to keep my dc safe.
But other than that, yes, they have playdates and sleepovers, but I do check in with them to see how it was after, gauge if they were comfortable, let them talk about anything that worried them, etc. I still let them be kids though.