I say this as a lardy person, in many cases you are not being unreasonable.
However, some folks have medical and/or psychological issues, or other priorities, or they're just excusing themselves or... whatever.
As part of the wobbly race I wonder wether the visibility of our problem leads to more comment on our problem from others which leads to more self justification. Or maybe we bullshit others justify ourselves because we know our weakness/issues are apparent to everyone immediately and want to get in with our excuses early because we are sure we are being judged, or, or, or...
There's a whole bunch of potentially self destructive habits that are not as immediately obvious (as long as someone is smart enough to cover up), drink, drugs, smoking, violence they don't tend to be as in your face as a size 24 bum.
I'm guessing here, you know someone overweight, they've said "I can't lose weight." You smelt bullshit but didn't want to call them on it. I found it similarly irritating when size 14 folks whined about being fat until I got to know a very thin woman with major self worth issues that included keeping her teenage figure. Each to their own hangups.