Yes, I see. Well, it's confusing because we tend to believe (and be told) that calories we eat are either burned off by exercise, or stored as fat, and it's not that simple. A protein calorie takes almost a third of a calorie to turn into fat or to burn as energy. A fat calorie only takes about 3% of a calorie. So, for a start, depending on macronutrient balance, we have different outcomes, re exercise/fat storage.
Then you have the fact that we have other ways of using calories than energy or storage. For example, if you introduce a calorie deficit, your body might keep all its fat, but be a bit less strict about homeostasis, so you might have bad moods or hot flushes or poor sleep or suddenly, spots. For example. There are hundreds of other functions your body can skimp on, before deciding it needs to use stored fat to compensate.
Then, there's hormones, like insulin and glucagon, which regulate your body's fat/sugar usage (put simply)
So, there's a hell of a lot going on. It might be that if you do less, you'll have more calories, so your skin will clear up and your hair will get glossy, but you won't gain fat. It might be that you'll pile pounds on in minutes.
Your best bet if you want to keep your weight down is to eat a glucagon encouraging diet, and to do that, you'll be minimising insulin release. Basically, drop your carbs lower, keep your fat and protein foods up, stop snacking between meals, and avoid alcohol. No extremes, just experiment a bit, and if you're gaining weight do it a bit more. If it does get extreme, and you're still gaining weight, up your exercise again, but play around with it. Your body is your experiment, set some healthy parameters before you start, and enjoy discovering what happens.