I grew up near a London premiership team stadium and I currently live near another London premiership stadium:
In both stadiums, the roads the stadiums are on get blocked off before, during and after the matches for pedestrians, so I wouldn't want to live there, but 1/3 mile is okay.
Parking on match day was a problem in my first home, so I just made sure I was home a minimum of 1 hour before kick off. Now the area is covered under parking permits, so it's not really a problem.
Traffic is also a problem, mainly after matches, but if I need to go out in the car, I leave before the match ends, or wait an hour afterwards. Traffic tends to be along the main roads, but you can get around using the back roads.
I don't keep an eye on fixtures, but I can tell when it's match day as you see lots of fans in Team colours starting to appear in the hours leading up to a match, so I plan around that.
Never had a problem with fans. I don't recall any violence or racism (I'm BAME and the local team had a reputation for racism). There is always lots of police and stewards about on match days, so I've never felt unsafe.
The noise isn't bad. If the wind is carrying the right way, I can hear the crowds, but very lightly and it's nice to hear cheering.
The businesses that get busy in match days are the local pub and take aways (chicken shops, kebab, sweet shop), so the chemists etc are fine.
I think living next to a stadium is fine, unless you live literally next to/opposite the stadium. It's not impossible to get around on match days, but you make life easier for yourself if you plan around it!!