Depends upon the accent and the way in which they speak.
I've found some accents easier to decode than others - I generally find Fante, Asante Twi, Sinhali or Konkani native speakers easy to understand compared to those whose home language is Kasem, Gujarati, or Tamil. Or a mate from Fife compared to the one from Shields especially when they're both three pints in. That's partly due to the tonal characteristics of their home languages and exposure to their speech (my friend from Fife just reminds me of my father, for example, and I've worked with many speakers of the first languages, so I'm used to the differences). Their pitch and rhythm of speaking matters - a monotone is often harder to discern meaning from, just as it is with native English speakers. Add in my Tinnitus and I can be losing important frequencies, especially if particular letter sounds I'm listening out for coincide with those frequencies/are altered in their accent.
However, there's also the physical cues - do they face me, do they move their lips clearly so I can get feedback from lipreading? Are they a mumbler in any language? Do they turn away or put their hand up to their mouth when speaking? Does their face move much at all - or have they botoxed any chance of my ever fully understanding away? Have they, from bitter experience, thought I'm just another fucking racist because I haven't understood them first time, hardened their jaw and become almost expressionless?
My worst case scenario is somebody with a strong accent with contrasting suprasegmental phonemes to English, high levels of non aspiration, allophones, neutralisation, background noise, botox, face mask or very bright/very dark light levels, has learned to speak English in a location that has a very strong accent, dialect and syntax/grammar different to those I have encountered previously AND suspects I'm just another racist dickhead. There's always a moment where I desperately hope that my saying 'I'm sorry', tapping my ear 'I can't quite understand' works - because I don't want to offend or hurt them. I just have multiple factors going on that mean with all the will in the world, the combination of all of these aspect can make it impossible for me to understand what a person with a particular native language or dialect is saying.