I'm kind of in two minds about this just because we had the opposite situation.
Whenever we went up to see PIL they would invite SIL and family over.
At the time it was me, DH, and our twins.
SIL has had kids every two years and at the time we were going up to see them she had five kids.
I'll be honest, we did get overwhelmed. We'd go up to see PIL and suddenly there would be an extra five kids around the place and PIL would spend all of their time with SIL's kids.
We dropped hints politely at first that we were finding it a little overwhelming and would like to spend some time with just PILs. It didn't really work so we ended up inviting PILs to ours and going on day trips etc because we wanted to build the relationship between our kids and their grandparents.
Your brother might just want to spend some time with his parents.
If you want to catch up with him, how about inviting him up to yours? Or suggesting meeting up half way?
I can see that you feel left out but at the same time it's reasonable of him to want to see his parents on his own sometimes.