His only reason for going would be to see me
I think this is the crux of the thing. There's a big difference between someone attending a public event and seeing you performing there, and somebody coming to a public event specifically to see you performing. Think of the emotional difference when you though it was the former, and then they say 'I only really came here to see you.'
Also the difference between performing in, say, someone's local pub where they go every night as a matter of course, or them following you round the country to see you in every pub you play in. It's just really really different, emotionally, and I don't think there's anything unusual about having a boundary with one and not the other. Many people feel uncomfortable performing in front of people they know, and if the people they know are worth their salt, they'll respect that.
I don't think it matters what the boundary is. It's healthy to respect it. You can't make yourself not have it, and more than you can say 'Oh look, a spider! Don't be scared.' to an arachnophobe. Well, you can, but you'd be insensitive to, if you knew of their phobia, and the arachnophobe would do well to stay away from you.
The only time this is damaging to relationships is if you can't do something that's vital to the other person for them to actually have a relationship. Like 'I can't talk about my emotions', or 'sex is a no-no.' But even in those situations, it's about finding the right person, who doesn't make you feel pushed.
I do let people in very slowly but I'm also open about it and why it's necessary for me
This is healthy, although I'm not sure why you're feeling the need to explain yourself to people. This is what I'm talking about with the self judgment. Nobody ever feels the need to say 'I let people in at an average pace!', because there's nothing to defend. You're feeling you have to explain it to people because you think they'll otherwise be critical of it.
What's wrong with letting people in slowly? Why not just do that, solid in your boots, and see who sticks around?