I sympathise with you, and have been on both sides of the fence (ie, struggling as well as more comfortable). I would be annoyed too, in your situation.
Now we are better off (on paper), we try very hard not to flaunt it amongst people we know, and are teaching our children to be grateful for our good luck, and to be considerate of their friends' feelings, as most are not so lucky.
My oh now earns a good amount, which we aren't used to, and so we're really worried about not having enough to pay tax, etc, as his salary varies (self employed). This means we possibly put more than we need away to pay it, just in case, obviously giving us less to live on.
We're also now paying a much larger mortgage than we used to (we had to move, and prices were very high although I know it's still a choice we were lucky to be able to make) and we have another massive debt we have to pay off, so our actual money to live on isn't what it seems to those on the outside.
I do feel a bit guilty that I'm better off than our friends/family, but we have it from the other side too (not saying you do this), and they often make comments about how easy life must be for us, or expect us to pay for stuff.
Most of these people still afford a holiday every year, whereas we've not been able to afford a holiday for years because once our money has paid for the essential bills/debts, etc, there isn't enough left to save for a holiday.
I guess what I'm trying to say, is that you never really know peoples' true situations, so shouldn't judge, and they probably don't mean it in a nasty way (maybe they're trying to empathize with you, albeit in a clumsy way).