BMI is not massively reliable - when you;re calculating weight in kg over height in metres squared (go the power of two) apparently it would be more accurate if it was somewhere between to the power of 2 and the power of 3 - i.e. height is underestimated.
Also, the healthy range may not be healthy or fully optimal for some - for example a young man with a rugby player type build can look quite unwell at a BMI of 18.5, very different from how they should.
I'd say it's gone too far if you feel weak, unwell, not as fully healthy as before (after a period of weight stabilisation), you don't sleep properly, you're not able to do your normal day-to-day activities, or you feel very cold.
I spent 7 months in an eating disorder unit, lost several friends to this. It's not something to mess around with. I'd rather be a few lbs above an optimal weight than below.
There's also the issue of needing a buffer - if you get sick in the winter and you're bang on your ideal weight, and you lose half a stone, you can't really afford to lose that.