It was breakfast or lunch, depending on what time everyone was up and ready.
You were the only one up and ready at breakfast. So you can't have breakfast with the others, because they're not up.
So, the 'or lunch' is now 'definitely lunch'.
But you've changed this by eating breakfast without them, and heading home.
If it were me,if I was wasting away and couldn't wait a couple of more hours to eat, I'd have just had some toast or something small so that I was able to still eat lunch with them.
It's you who changed the scenario.
I am also confused as to why you agreed to lunch, as it sounds as if you definitely wanted breakfast as wanted to go home earlier than lunch?
Why not say last night 'If you're up I'll have breakfast with you, but I won't be able to stay much later than that so can't join for lunch'.
That might have even changed their plans-they may have made sure they could be up for breakfast with you. They expected you to be okay with either meal, because you said you were.