I've given up trying to rationalise their mad thinking tbh. Dh's ex is probably loving the whole situation, as it helps her to separate him from his children...
The basic mistake you made in your reasoning was to view me as part of the family - you see, dh's ex is their only sister in law, so of course they invite her. They can't invite me as I'm not their SIL, and so by extension our dds are not as valid (for want of a better word) as my stepchildren are - they seem to take the view that dh's 2 children are the only true relatives, are somehow a "better" neice/nephew because they were born first, conveniently ignoring the fact that the dds are as related to them as my stepchildren are!
They really are truly bonkers, and most of the time I am able to laugh at it dh's older brother has never even met the dds (he does live abroad so more difficult, but there have been opportunities, most notably when we offered to drive 200 miles, with dd1 aged 7 months, for lunch as they were too busy to visit us on one of hteir trips back - they still said they were too busy, so we've not bothered offering again.) Dh's younger brother refused to speak to me at all when we last stayed with them - we were there for 4 days, and he did not manage a single word - more than a little awkward, especially as at times we were the only 2 adults in a houseful of children (his 3, my 2, all under 7)
I am a great believer in people knowing, on some level, that they are behaving unaceptably. At some point, your PIL and grandPIL will all realise that they have behaved appallingly. I know this does not help you much, but can you imagine feeling how they will feel at that point? I'm not explaining this very well - it's kind of like the Christian idea of confessing all to St Peter before entering heaven - at some point your dh's family will know, and have to face up to, the shame of having treated you the way they have. They probably do know already. On that basis, it can't be too comfortable being them, can it? They can't be very happy people, to treat people around them like that.
This is the kind of way I look at it, to deal wiht the obvious bitching that goes on towards us from most of dh's family - it helps me a bit. FWIW, I'd stay right out of any further contact. Would you mind if your dh (if he wanted to) saw his family without you? This is the kind of thing I'm contemplating. If my dh's family don't want to see me, then fine, I'll stay away - it gives them less ammo to take to any bitching sessions. I have a perfect excuse in that I can claim to stay away for dd1's comfort, and if all that gives them is a way to pity me for my "difficult" daughter, then they can go right ahead, would give me pleasure to think they are gossiping about something so blatently untrue.