Cutting out starchy carbs altogether gave me less energy, a slow metabolism and not enough fibre in my diet, even though I was really careful with nutrition generally when I did so. Also it made me absolutely crazy and want to binge on the worst kind of processed carbs and also alcohol as it caused a horrible imbalance and really low blood sugar, and I'm never a binge eater or drinker normally. Also I just felt like I was eating too much protein - my gut does not cope well with a high protein diet and most people would do well to eat less, not more protein- I think very high protein is absolute bollocks, and actually does damage to your body. I lift weights and get enough protein just from my normal diet including meat, fish, cheese, milk and tofu without any nonsense supplements.
I'd just swap the more refined ones for whole versions where possible and portion carefully and see how you go. I do a lot of exercise and absolutely nothing beats starchy carbs for energy for me, plus they are so comforting, filling and convenient.