I have discussed this a lot with my husband. He used to moan that I would eat the same as him plus loads of chocolate but never get fat, whereas he always puts on weight. But he literally eats twice as much as me and then picks the bits off my plate. So we're eating the same thing but he has double my calories.
I also walk everywhere. I hate sitting in traffic so I never drive. I walk to the shops and carry the shopping back, which my husband thinks is mad, but it's only a 15 minute walk and I like it. (Stops me impulse buying stuff too.)
I also hate feeling overly full. Satisfied, sure, but if I ate an entire portion of restaurant food in the UK, I'd feel sick. Conversely, when I'm in Japan (have family there), I can eat a whole plate of food and it's just right. The portions there are far more sensible (and the people far skinnier).
I think for so many people, they feel an emotional attachment to food. I know my husband does. When he's sad, stressed, tired, whatever, he eats. When I feel like that, I have a cup of tea. Not because I'm denying myself food, it's just not what I want when I'm feeling down. When he goes on a diet, he'll note that the first few days he's starving, then he gets used to it. I think for a lot of people, that feeling of being starving is really unbearable so they can't lose weight because they feel they have to eat.
People talk about metabolism and so on, but I just don't buy that. Of course people have different metabolisms, but more often, they are just lying to themselves. I read somewhere that obesity in the UK has doubled in 20 years. There's no way people's metabolism has changed, but eating habits get worse and worse, and people move less. We didn't have all the chain coffee shops, for example. I used to sometimes buy those shitty Starbucks types drinks until one day I looked at the calories in them - I really don't count calories, but those things are a full meal! It's just nonsensical to drink them if you care about your health on any level.
I used to teach these Korean guys who were going to the US, and one once sent me a voucher for Starbucks. It was for me and my husband - for two coffees and one piece of cake. I think that says it all really. To them, a piece of cake is for two people, whereas I think we'd almost always buy two. As a result, they're almost all skinny.