I can find their behaviour annoying, without necessarily finding the person themselves annoying IYSWIM.
I have a relative who has crippingly low self-esteem. It manifests itself in anxiety (she's not a relaxing person to spend time with - constantly tidying, never does small chat, always worrying about boring other people or putting them out in some way), doom and gloom (she's always expecting the worst) and always putting herself last (overly polite, "no no, it's fine, it doesn't matter, you go ahead").
She's been on ADs and anxiety meds for year, has had counselling for years, but nothing seems to break through this utter conviction that she is boring, annoying, unattractive, stupid and generally at the bottom of every pecking order. She's been bullied in every job she's had - I think her cringing manner brings out the worse in some people. Even I find it hard not to get impatient and a bit brusque with her!
But she is such a lovely person, considerate and kind and funny when she can just relax for five minutes.