YANBU. DH and I have got this over the years too. We have not had an easy life by all means, and have had many ups and downs. Right now we are OK (and have been for 5-6 years.) No mortgage, can afford to go out when we like, and both work part time - me 24 hours a week, him 28 hours. We have a little cottage near woodlands in a low-population rural area, and a 67 registered car.
DH was told by a colleague a few months ago, that 3 of his female colleagues had been saying 'how on EARTH can he work part time? They have holidays abroad, lots of daytrips inland (4 or 5 a year!) and a 4 year old car, where on EARTH do they find the money?' They don't think for a SECOND that we don't spend anything other than on essentials, don't smoke, don't have takeways, only go to the pub once a month, go on holiday abroad once a year (cheap package deals.)
We also live in a small house with low bills compared to many. No mobile phone contract, just £10 a month for unlimited texts and minutes and 10GB of data. No sky tv, no designer clothes. Kids have left home and are 100% self sufficient.
Also, what the FUCK has it got to do with them? People can deny it, but they are 100% SO jealous!
And some ten years ago, my very good friend and her DH were on the brink of financial ruin as her DH lost his job and they'd just had a baby (and already had a 2 year old,) and he had to take a job with half the salary of his old one.
They were filing for bankruptcy after 2-3 years of juggling finances, and robbing Peter to pay Paul. She also lost her mother to cancer, just 7 weeks after she was diagnosed, and her brother was killed in a car crash 2 weeks after her mother died.
All the strain (including looking after a small baby and a toddler,) had a big effect on her marriage, and she and her DH almost split up. Her dad had died when she was 17, so that was her mum, dad, AND one and only sibling gone. And she was only 31!
She was in a local community hobby group, and several single mums there who lived on the big estate a mile from her, looked down their nose at her and called her a yummy mummy, and said (behind her back) that she was 'privileged,' and how it was right for HER having a husband to support her. (She worked 3 days a week herself!) And all they talked about was how poor they were (whilst smoking 20 cigs a day, and getting pissed at the pub 3 nights a week.) They were really nasty and bitchy. These were women in heir THIRTIES.
Just because she never complained, they assumed she had a perfect life, not one that was falling apart around her.
People who look down on you and assume you're 'privileged' because you never complain about it are obtuse, narrow-minded twats.
Also, in my experience, the more people complain about something, the less they have to complain about.