Why are you proud of your grandfather's role in the British Colonial Service in Tanzania, specifically?
I understand you thinking of him as a lovely person in general. But to be proud of the work he did for a particular organisation, you would need to know what that organisation did, what its aim was, what the political context was at the time.
And, as you are choosing to declare this pride now, in the current political and social context, you need to know what the legacy of its work has been.
What did you find out about that on your trip? About the legacy of that organisation? About how Tanzanians perceive it now?
Pride is a very strong, very specific thing to feel. You feel pride for someone's achievement, not for them just being themselves. That's liking, or general admiration of them as a person, rather than admiration for what they did.
If you declared yourself proud of your grandfather's role in a colonial organisation, it is not unreasonable for your friend to have understood that as you declaring support, now, for what that organisation did then and for the continuing consequences of that. If that organisation did racist things (which given its name and place seems pretty likey) then....
Think more, know more, before declaring your 'pride' in what someone has done.
Pride is a very strong word. I think you were misusing it to mean something else, that didn't relate directly to actions and achievements. You and your friend may therefore have been talking at cross purposes.
I do think there's more room for that linguistic misunderstanding now. Pride has become connected to identity, used as the opposite of ashamed, as in gay pride, out and proud. That detaches it from achievement. It also strengthens its link to identity and to political declaration.
That is what your friend heard - you aligning yourself politically with a colonial identity.