Interestingly, we had this conversation today. My sisters first born passed away and we always go to see him with her two DC and my niece and nephew.
They love going to see him and are 2 and 4, but naturally they're not necessarily the most respectful but obviously mean no harm.
I did see a gentleman look at us today like we were doing something wrong, and I actually understand because he obviously is grieving, but it's definitely not unreasonable for her to take her children to see their sibling.
So yeah, YABU, but I can totally see why someone might not be impressed and I think that's fair enough under the circumstances.