I think if it upsets you that you are a late addition then there is no earthly reason to feel you need to go.
But a breakfast sounds as though it is a reasonably small number invited? If you felt like joining, I wouldn't decline simply out of pique.
But no, there is no reason you have to accept at short notice, so maybe don't if you are upset.