It was rather silly of your sister to NOT insure her wedding tbh - it's possible to take out policies that have nothing to do with travel agents etc and if she knew the relationship was in trouble, it was short-sighted of her not to do this for her own good. But that wouldn't have helped her guests, as I don't think ther costs will be covered unless they were part of the wedding party.
I have to say, I also think it was short sighted of her to forge ahead with wedding plans of this nature for an uncertain relationship. And that I might well feel annoyed if I'd spent money to go as a guest, only to be left out of pocket with nothing to show for it. It smacks a little of the 'bridezilla' not thinking about anyone but herself.
When were the invites issued? And how long have they been having problems? If there was any sign of issues before the invites were issued (or even reasonably after them) then she (and her fiance) had a duty to warn people not to commit financially. We don't know the circumstances of the split, what portion of blame lies with your sister etc, but her friend's will and that may well be making a difference.
I'd feel very different towards a friend of mine who'd cancelled, say, because her other half was suddenly caught having an affair that she'd known nothing about, than towards one who had clearly been having mutual issues for months/in counselling/etc. One is clearly not her fault - the other, well, I'd be wondering how good a friend she was to let me risk all that money under such precarious circumstances.
And, yes, if she did know there were possible problems, she did have a duty to inform her guests. I had to warn a couple flying from America for my wedding when my MIL had a stroke six months before we married, purely because we had some doubt over whether we'd have to cancel. If we had, and they'd paid, I'd have felt obliged to refund them - even though I, personally, could have done nothing about the circumstances.
All that said, it is in extremely bad taste for the guests to have asked for the money. They did choose to go, and they should have covered themselves. So you are not unreasonable to be annoyed with them for it.