Lost in your shoes I'd bail at this point. I'd like to pretend I'd be generous-spirited enough to understand about his severely disabled child but, quite frankly, I'd resent the fact that my partner was unavailable for outings every weekend and after six months or so I'd probably be channelling anger and general pissed offness.
The other kids I'm less fazed about and if you're happy with a man that's learned from some life lessons then your family will come round. He can't be judged for a ONS and he can't be judged for not knowing he had a child.
As for earlier children I'd want a man who was in my life to be in his child's life once he knew about their existence (provided the child wanted that) so I'd judge negatively if he didn't have contact where it was welcome.
As for 'haunting' what exactly do you mean by that? Money, guilt? He seems pretty adjusted to stuff so is it more about them haunting you?