I don't think anyone is saying eccentric people are nasty, or men and women can't be friends, or that there can't even be a frisson of attraction in platonic friends, or even ex-partners can't be friends.
It's just not an uncommon experience the OP is having, hence all the voices.
I think a lot of women, when younger did think "oh he's some lonely quirky nice guy. And we have lots in common so we can be friends even if I'm not sexually attracted to them"
And we went through contortions so as not to "lead them on" or make sure we weren't just using them, and didn't give them false hope and encouraged them to pursue other women.
In other words, trying to respect and value them as we would a genuine friend. There are some great platonic male/female friendships out there, it would be a shame to live in a society where they couldn't arise.
Trouble is, these "lonely quirky nice guys" aren't always some heart of gold in an unattractive exterior.
They can be very manipulative and turn very nasty if they don't get what they want (and inevitably, the women they feel entitled to are much better looking than them).
Instead of making a clear decision to just go "sorry I can't be friends" if they find it emotionally hard, they hear "let's just be friends" as a green light to cause drama and manipulate.
This guy: he's basically demanding OP invites him to her friend's party, but in a very passive-aggressive way.
A single mother to a baby and he's asking to "look after and be a father figure" to her baby who he hasn't met? It's like he's using his social ineptness as a weapon to get what he wants to guilt the OP into letting him invade her life.