it's possible it would make it worse. I have several, one of them faces a road, and you can hear traffic. It's worse if I take the fan out of the duct. Through the others I can hear wind in the trees when all is quiet at night, it is not noticeable during the day.
The modern quiet fans pretty well fill the cavity with motor and fan, and have a shutter to prevent backdraught, which makes some difference.
Badly sealed holes in walls (usually around replacement windows) let in draughts and noise. Once built, it's difficult to improve the mortaring (unless the gap is big enough to get your fingers in) but not so difficult to use expanding foam and silicone sealant.
If you have a cowl, you should be able to see, from underneath, the big flap swinging open when the fan is turned on, and it should blow tightly shut if wind is blowing against it and the fan is off. They are made with a plastic ring in the surround, that should be a close fit in the plastic duct (tube) through the wall. If there is no duct there will be no seal. The hole in the wall should be made with a 110mm approx core drill, which makes a very neat, round hole and cuts out a cylindrical plug of brickwork. Plumbers and builders will usually have one, it also fits soil pipe, and a smaller size is used for waste pipes.
In the old days a hammer and chisel and small drill were used, which makes a more ragged hole.