I think people have very contrived notions of what love is and it makes for a great deal of unhappiness.
You don't 'fall' into or out of love with someone. Because real love isn't like that. Nor does it have to do with all this fancying nonsense. It is a continuum and a spectral disorder, if you will, that can wax and wane but you know it will endure.
Because REAL love is when it doesn't matter if he turned into the Hunchback of Notre Dame after a car accident, you would love the person he was inside. And if his physicality put you off, you would be willing to tell him that and try to work through it, rather than 'Oh, I don't fancy you anymore, the 'love' is gone' (because you got old, fat, sick, bad teeth, etc.) because you respect him.
And he would do the same for you if the shoe were on the other foot.
If such is the case, then yes, often enough, you're better off without such a person because you're not doing either of you any favours.
I'm not talking about partners or husbands who behave abusively, have substance abuse problems, drink or gamble all the money away, etc.
But no matter how it works out, do yourself a big favour and get some counselling and work out how it feels to really love someone before getting involved with anyone else.