The one I don't think I will ever have is a husky or other wolf-like dog breed. I think they are so lovely, but;
Their prey drive: I have a cat who is my little daring and SHE COMES FIRST. Any dog that might eat her is off the list.
Their size: I currently own an elderly Lhasa Apso (a pretty common breed over here!) and her bones are failing her. It is so helpful to be able to lift and carry her anywhere she wants, I don't think I would have a dog I couldn't pick up and carry at least to the car and back. Which limits dog breeds to 10kg, 15kg at a stretch.
Their energy level: I'm a couch potato with short bursts of energy. While I would be willing to run for say, 1 hour every day with a dog, a Husky's endurance is just too much for me. Now if only Alaskan Klee Kai's got more popular and standardised, that would be swell.
The Miniature American Shepherd
I love Aussies. Their appearance, personality, everything. They just happen to not be quite carryable. So when I heard of Mini Americans (in 2016, I think, they had just become a breed in the US) it was perfect. I just don't have the time for one right now, but I dream of getting one by 2030... so for now I follow MASCE, AKC and some lovely kennels around the world to see the development of the breed.
Finally, the Cavalier.
Everything is perfect about them, except the mitral valve. I don't think I could have a dog that would have close too 100% chance to have a heart disease by age 10. It would feel like a ticking bomb for me. So I eagerly wait until researchers fix it. I would even be fine with out breeding for a couple generations. Funnily enough, the elderly Lhasa Apso I mentioned earlier does have a faulty mitral valve, but differently from Cavaliers, hers is still subclinical at nearly 15... It's the bones that are her undoing.