Regardless of who's "right" or "wrong", the practical situation seems to be that (1) your H wants to visit his brother/SIL overnight on a regular basis, but (2) you no longer want to go and have previously stayed home while H and children go. Now (3) your D doesn't want to go unless you do.
If this were just a holiday, I'd say each adult has equal say in destination and if one person says no, it's either off or that person agrees to sit it out. With family, though, it's a bit different - make sure you listen to & understand your H's feelings about wanting to go to his brother's and wanting the children (and possibly you) to come along and have a relationship. It can't be forced, though.
If D's old enough to stay home even though you're working, I'd let her stay and let your H make an excuse. The other option (probably preferable, but harder and more time-consuming) is to have an honest discussion with your H and D so he understands how she sees the situation and what she needs so that she's comfortable going. If your H is willing to protect her in the way she apparently feels you would, and she is convinced of that, then she can go with him. Of course, if her (and possibly your) objections are part of a misunderstanding or can be solved in a different way, that could come out of the discussion too, but it has to be consensus not one person overriding the other(s).
I don't see the point in having a row about it; that won't lead to a solution or compromise.