Hmmm, at the risk of being torn apart... here's my take.
You're probably right to read something into the wink. Everything that happened in that exchange was perfectly normal, happy, family type stuff (resembling in many ways the interactions my wife and I have). Up until the point where he walks out... he'd probably really enjoyed his visit with his kid and you and felt he was doing something meaningful by helping out and washing the pots (you're hardly describing a scumbag here.
He may well find you extremely attractive (you are, after all, the mother of his child) and be inwardly feeling a complete prat for ever letting you go.
You have got a child, who you don't want to mess up, and a lot of other things going on in life. The wrong thing to do is to let anything happen on an assumed basis (EITHER "accidentally" sleeping with him OR driving him away out of fear).
One time when he visits... preferably whilst he is doing something like cleaning the toilet/washing the pots (cleaning the toilet is better for this... men will then keep their hands to themselves so you can finish saying what you want to say without ahem distractions, and are also unlikely to run down the street in marigolds if it goes the other way)... have a serious chat about possibilities WITH RULES. So "I'd like to see more of you, but that means XXX", "If you don't do XXX you're out". Finish with a line like "we're not kids any more, we'd have a responsibility to do this right." Then put him on probation for a couple of months (no sex, but quality time together). Honestly (speaking as a guy), you'd be making an offer he couldn't refuse... he'll either say "we're best as we are" or "I'd like that" and either way you'd know.
Be an APE... Assess, Plan, React!