I can't find any direct evidence that this would be harmful. There is research showing that a build-up of smoking residues on furniture / car interiors contains substances which can be harmful, but this has been raised in the context of the home environment with a build-up of smoke, not clothing worn for a few hours. I have not seen any suggestion that very short-term exposure to the clothing of a smoker could be harmful. See www.nhs.uk/news/pregnancy-and-child/concern-over-third-hand-smoke/
In addition, we know that smokers' breath can contain higher levels of carbon monoxide and this may explain part of the increased risk of SIDS when a smoker co-sleeps with a baby, but I have not found any reputable healthcare sources suggesting that one-off short-term exposure in an open room might be a risk. The situation where a smoker parent is breathing over a baby all night long is clearly quite different.
Realistically, as others have said, air pollution is a much more apparent threat. One study suggested that babies of non-smokers living in a polluted part of Scotland were only marginally better off than babies of smokers living in cleaner air.
"Lead scientist Dr Tom Clemens explained that his team’s findings showed that “a foetus with a non-smoking mother exposed to high pollution levels is only slightly better off than one with a smoking mother exposed to low levels of pollution”
www.euractiv.com/section/air-pollution/news/air-pollution-almost-as-bad-for-babies-as-smoking-during-pregnancy/
That's talking about parents who are actually smoking while pregnant, not just holding a baby for a few minutes.
Realistically, the risk to your baby from this is going to be too small to measure. As others have said, if you're really worried about minimising risks then perhaps moving to the countryside would be something for your family to consider. But let's hope the visitors don't ask to hold baby, so you don't have to worry. Good luck.