Oh this really struck a chord with me. You have my full sympathies!
I don't think you should threaten to stop contact, though. Your son probably needs and wants to see his father. Also, if you over-react now, your son won't feel able to tell you about anything else that concerns him at his dad's.
I think the only thing you can do is try to reason with his dad. At the very least, even if it's not doing him any harm, it won't be doing him any good, and it's not a good role model for him either. Maybe you could also suggest that your son goes out of the room when his dad starts smoking, or that your son, himself, asks him to stop (although that might be quite difficult for him, and you don't say what age he is). If you are reasonable, your ex might be reasonable too, and alter his behaviour, even if he doesn't admit it to you.
When you split up from the other parent of your child, you have to accept some things, keep your nerve and pick your battles. I know this only too well. My ex-husband used to do all sorts of things that I considered highly dangerous when he was looking after our children, and I worried myself to a frazzle about it. They both survived their childhood, however.
I feel I can say this to you as I really do understand how you feel. I have no idea if the decisions I made, when I was in a similar situation to you, were the right ones. All I can say is I never minded people smoking until my daughter (aged 37) was diagnosed with lung cancer. Obviously, it's really difficult to say that one thing is the sole cause of something else, but her grandparents used to smoke in the house and car, and it might be possible that their smoking had something to do with her illness. But even if it didn't, it was still unpleasant for her, and she complained, many times, to me. I did nothing, and, to my shame, even told her to be more respectful when she complained to them.
However, her brother had to put up with it too, and he didn't get lung cancer. Also, there are many more environmental causes (diesel engines, for example, according to my daughter's consultant).
I don't know how helpful this is to you, as looking back I don't know if I should have acted or not. You will just have to rely on your own judgement. Good luck!