We live in Twickenham (south) and match day traffic is terrible!! But its only a couple of weekends a year (I think 12 ish) and as long as you're aware of it, its very easy to plan around - its only in the 2 hrs before a match and the 2hrs after that it is a problem. Generally we don't mind at all.
There are a couple of roads (Varsity Drive opposite the stadium for instance) that are totally closed though so you can't get your car in or out - that's a bit more annoying.
There are a couple of match days (Army v Navy and the Sevens) which are bloody awful and we generally try and get out of Twickenham on those days - the drunkenness is out of control, pubs are forced to shut at 8pm to try and control it but you'll still see people vomitting/wetting themselves in the streets.
Modgen - yes, you can smell it sometimes. But the prevalent wind takes the smell North over Isleworth so its rare we ever smell it (maybe once every couple of months). Never been bothered by it though, its a minor thing.
We do have a couple of mosquitos to squish on summer days with the doors/windows open - but we've never knowingly been bitten by one.
The flight path is also annoying. Generally planes are coming in to land over Twickenham which is much quieter than taking off. But every couple of weeks they take off over us and that's a pain. We installed soundproofed windows (secondary glazing) in the toddler's room because they would wake her up (cost £500, best money we ever spent). But we aren't bothered by them and I can sleep though them - unlike when we lived in Putney and they woke me up at 4.30am every single day.
But those are the only bad things about living there, we've been in the area for three years and generally I love it. The commute is easy into London, I've made some excellent local friends and there are a ton of nice services for families etc. We have no plans to move...