There are so many places in London that are near some kind of major event venue though... O2, Excel, 10+ football stadiums... it's hard to be anywhere well major events are not held, or tourists don't go.
Millwall stadium generally doesn't hold that many events. It's literally a few hours of the area being crowded for a couple of hours a week.
Transport links are excellent down there and it's surpringly central.
If it would be a team in a city like Leicester, Brighton etc I might think again, but in London, unless you're well out in the suburbs, you'll never be out of the way of any and all events. E.g. if you are anywhere near a 2-mile radius of Arsenal, you'll be somewhat affected by match day, whereas at Millwall it'll be much less.
I live a 10-minute walk from a Premier League club and it's minimal disruption, we have permitted parking only, you'll likely have the same.
You'll have a road closure maybe 45 mins before the match, and 45 mins after, but to be honest, I have been driving through directly past the stadium even on match days close enough to kick-off, and it's been fine.
Millwall doesn't have that much going on in recent years, and that's unlikely to change.