It isn't your body "holding onto everything it can" but smaller lighter bodies not actually needing that many calories to do things.
Use an online BMR calculator to find your basic calorie requirement to maintain your existing weight doing literally nothing. I suspect a 5'2" slim person would need around 1000-1200, mine at 5'5" and 9st is about 1400. Any activity will then need added calories but the much quoted 2000 per day calorie requirement for an average woman is too high for the "average" woman these days, because of sedentary lifestyles with too little exercise, it is very misleading. I also agree with nigglenaggle, i think people have a perception that exercise burns many more calories than it actually does.
Don't discount the benefits of exercise that seem to burn less calories though -- weight lifting for example, will burn much less than running while you are doing it, but the muscle you build will burn many more calories at rest than fat, so boosts your metabolism full time. Yoga/walking keeps muscles active and helps prevent injuries/illness-its all good so try not to focus purely on burning calories!
I did a sponsored fit thing for Children in Need last year; 45 minutes spin, followed by 1 hour body pump, then a 1hour body jam class, 1 hour body balance, 1 hour body power finished with 45 mintues circuits. I only burnt 1500 calories. Is surprising how little you do burn, the body holds on to everything it can!
I cant understand why the calories Ive burnt at the end of the day are so low. yesterday i walked 17500 steps whilst out for the day. my calories burnt was only 2100. I'm not very tall, 5'2" but it doesn't seem like very much after all that walking every day im surprised at how little calories i seemed to have burnt. i set my details on on the fitbit website.