Point 6 is obviously of huge concern, but if that was what was stopping ex taking his son home would he really still insist that DS was allowed in the company of this person ? If so .... and this goes to court, then you would need to express your concern and it's likely he would be ordered that they are never in each other's company. Do you really think your ex would place DS at risk like that ?
All else, I'm afraid, sounds like sour grapes. Yes, of course it's extremely shitty when you are left in the lurch as a single mum and 99% or more responsibility is left to you. However, that is not DS's fault, and by punishing his father you are also, most importantly, punishing the child. My ex too moved away, had to be nagged into regular contact, there've been various money issues, he's been thoughtless and so on .... but all the while, what I wanted to preserve, as much as possible, was my son's self esteem. I didn't want him (my son) to feel bad because he had a bad dad and therefore did everything I could to encourage contact - even though there were times when his idea of parenting wasn't the same as mine. In the bigger picture, those things really aren't anywhere near as important as a child who feels safe and secure in the knowledge that both parents love them, and that they never have to "choose" between either.
Bedtimes, pick-up times, return times etc can all be negotiated. Ditto safety issues if done in a tactful way. How much better would it be if you approached this with an apparently open mind and told your ex you weren't adverse to the idea but wanted some reassurance from him "I'm probably being silly" about how he'd cope, rather than taking an "over my dead body" approach which is bound to get him on the defensive. I have to say that you saying contact's okay so long as it's local does give the impression that you want to monitor contact and that it can only happen the way you want it to. That attitude's likely to be quite demeaning and offensive for most 40 year olds and if you stick to your guns things could get horribly contentious - with poor DS stuck in the middle of it.