I don't think anyone's in a huff with Nigeria. Are they? What for?
I think the issue is that Harry went to Canada and purported to make representations about reparations to First Nations people (knowing Canada is a Commonwealth country, he's a Prince, and once held a fairly important role re the Commonwealth before he resigned his royal duties) in a quasi-formal setting (at least as far as the First Nations representatives were concerned) when he had no authority to do so. It was outrageous, in diplomatic terms.
I don't believe anyone has an issue with Harry taking Invictus to Nigeria and doing whatever IG-related thing he wants to do there. But the Invictus Games aren't publicly funded, Harry isn't a representative of the British people/military/government/royal family. He has no standing to represent Britain. This is probably in response to early reports in Nigeria of him and Meghan going there to conduct their first "official tour".