I'm on the fence. I've never been very fat or very thin , but I've been a good size 16 and am normally around a ten, and I'd say the only difference was I became more invisible to men when I'm heavier than when I'm slimmer.
Are people judgemental, yes I think some are, if they perceive you as very fat or very thin and I reckon both sides get their fair share of negative comments. However everyone is different in terms of what they view as very far or very thin.
Do I judge, to be honest, yes sometimes, if someone is excessively over weight, or excessively thin, in my mind the thought flashes through that it can't be healthy but it stops there and doesn't impact the way I treat the person and I'd never dream of commenting.