Anna, I don't think you are being unreasonable, there are always going to be children we can't stand that ours want to befriend.
DD'S best friends are so totally opposite, one is the child her teacher told me she wants to discourage her from sitting with in class as she is disruptive, the other is the daughter of one of the parent governers, who is also a dinner lady and a school helper, and often has people from their class over/goes on playdates.
The 'troublemaker' is always extremely well behaved at my house, but I ahd other parents say to me at DD's halloween party 'Oh, my you invited ***, I wouldn't have her at one of my parties' , the other child has cried over the slightest thing everytime she has been to my house, complained that DS is here, but won't go in a different room to him (expected me and him to go out of the living room do they could play in there!!!) and is basically a little whingy brat.
I don't even feel comfortable telling the whiny one that what she is doing isn't nice to DS, in fear of her crying, but I'm such a sap when her mother turns up, saying she has been ever so good
I wish my DD didnt choose her as a friend, but there isn't anything I can do, really.
I think three is quite young, but my DS is the same age as your DD and I know he'd be quite hurt to be called names, so I'd probably just say 'we don't have name-calling in this house, it's not nice' and leave it at that.
So no, wanting to discourage it isn't unreasonable, but if you were to do more than just eek out contact at home then I think it would be unreasonable.