I agree. Invictus has royal connections and is also linked to the military, with perhaps high ranking officers in attendance. So it might well be considered a high risk event, with or without the Duke of Montecito. That doesn't mean he's going to get his own security detail to swan about the country at will, which is probably what he wanted.
There's no reason for Meghan to be there since she has no connection to Invictus other than being married to the Founding Patron, so I doubt she will attend.