I am a man who had an affair that destroyed his first marriage. I have also had partners who cheated on me, so I've seen it from both sides. Everything you say rings alarm bells for me. I've said those things myself.
I used the "don't want to lose her as a friend" justification to myself in the run up to the affair and I know one of my partners used the same justification before cheating on me. Believe me, that is just a way of telling yourself it is ok to hang onto something which, for the sake of your marriage, you must end. Now.
This man is a colleague. You must treat him as such. Nothing more. That may upset him if he is attracted to you in the same way but he will get over it. So will you. Maybe one day you can be friends again when all the feelings have gone in a couple of years or so. But not now.
There may be a crisis when you stop the friendship. He may open up about his feelings for you. He may tell you that he doesn't want to lose you as a friend and that you can just be friends without endangering your marriages. If he does, he will be wrong. You must be strong.
It is too late to think about strategies for defusing the situation. This has already gone too far. How do you think your husband would feel if he knew about the lingering hugs?
I'm sorry but you cannot be friends with this man. End it. Now.
If you cannot end your friendship you will end up having an affair with this man. So you must choose. You can have this man as a friend or you can keep your marriage. You cannot have both. If you try you will end up seriously hurting your husband and children, his wife and his children.
It is this man or your husband. I'm afraid there is no other way.