Absolutely agree.
I get pestered when I'm slim. When I'm fat, as I am now, I am sexually invisible....but that's no bad thing.
Perception-wise, Fat = lazy vs Slim = in control and together. That's my experience anyway.
That being said, I find people like fat me better than slim me, both men and women. I seem to be more approachable and less threatening. Married men would avoid talking to me as a slim (f*ckable) woman but no longer do. Could be an age thing too though, I suppose more 40's men are married than younger ones, but I've noticed the difference in the talk nowadays, they're not trying to pull, they're asking things about your kids etc, normal conversation with no undertones same as I'd have with a woman.
In truth though, no-one cares that much about other people being fat. Of course you're judged, but ultimately people are insular and more concerned with how they are being perceived themselves to worry about other people's weight.