If you're rich, yes, you can afford to evacuate before a storm (although many choose not to). The thing is, you almost never get more than 1 or 2 days notice that a storm will actually hit. The few flights that are still on, will be ten fold the price and you need a US Visa as vast majority of flights from the Carribean go through Miami, and most people don't have a US visa. But because storms don't always hit, most people ride it out in a shelter and then take a flight out if it was extremely bad (you don't want to stick around for more than a day or 2 in a place with no running water and the looting invariably starts by day 3).
There is a big divide in the housing. Middle and upper class people will have proper concrete houses, raised at least 5 feet off the ground, and not in a flood area. So, as always, poor people get hit the worst. A well built concrete house will not get destroyed, even by a Cat 5. The roof will get blown off but the house will still be there. Poor people living in shacks or extreme flood area will lose absolutely everything.