There is no easy solution to this. My mother smokes and when the children were babies she did go outside and we would open windows and doors, but it still felt smoky. It's not the smell so much as the residue.
As the kids got older she became less consistent about it, and would sometimes just stand in the doorway....it caused a few arguments.
Ultimately we did go round a couple of times a month for a few hours at a time....its important to maintain family relationships and she did help out with childcare. As the kids got older they did spend a bit more time there. I wasn't very happy but when I was a kid BOTH my parents smoked in that house, while I was in the same room, and while I am not saying that safe or advisable I am pretty much OK, as are lots of others.
So, I think occasional visits would be fine, but it's absolutely right that your OH should state very clearly that he is to smoke outside (10 steps from door) while you are there, and for at least 30 mins before, and they should air the house/rooms you will be using before you get there.
Also advise them airfresheners are not the same. My mum drowns the house in airfreshener thinking it disguises the smell but it just adds more chemicals to the mix!
I don't think it's reasonable or even effective to say to a smoke don't hold the baby, but take your own play mats, feeding stuff, toys etc, don't let him suck or chew on anything that has been in the house for more than a day.
You are a stronger person than me to have held out this long, I wish I had been more forthright, but I don't regret letting the kids go to their grandparents.