I adore all four of my kids equally, but those saying its cruel are daft! Or had horrid grandparents! lol.
There is tons of time to take trips as a complete family, if this will benefit your older children, who, sadly, will one day be wanting to have their holidays with their mates not their parents, and give you guys some time to do the things that are so much harder to do with the little ones, thats great! It is hard to do things like go swimming when you have a tiddler, doesn't mean you don't want her in your lives, but if you have the chance to have family take care of her while your older children get some attention, thats healthy!
Its not like you are dumping your 2 yr old in a childrens home, laughing in their face as you and the other kids bugger off to have fun....your 2 yr old will miss you a bit, and you will miss her, but thats healthy, and fine, and does absolutely no damage! If you were talking your seven yr away and leaving your 6 yr old at home, different story!
Your kids will all have a good time no matter what. And I would tell that gf of your brothers to keep her nose out! Have a great trip no matter what you do!
Ps I did this, used to take my older boys away and leave my 3 yr old and 18 month old babies with their Grandma, and they loved every second, and would cry when it was time to come home! Now my boys are older and doing their own thing and going on holiday with girlfriends and mates, and my girls now get all my attention, with some wonderful memories of being spoiled by Granny! ;-)