Depends how far they have travelled I guess, but your op suggests it’s a local friend for dinner rather than a visit that involved travelling or coming to stay in a formal sense.
Even if it was the former, and they were coming for the weekend, it’s usual to leave after lunch on Sunday.
So for local friends staying overnight I would have thought that pretty promptly after breakfast tbh so by about 10 am ? Certainly no later than 11am.
If the friends linger and exploit your hospitality, then unfortunately you have to be explicit and say you have “plans” , but you need to do that the night before ideally, as it comes across as rather rude in the moment unfortunately.
Some people just don’t have any sensitivity unfortunately, or basic manners, and if they don’t host themselves, little idea of what it involves,
I get this often nowadays with simple lunch or tea invitations. One person I invited to tea talked non-stop about themselves for four hours straight without pausing for breath and with presumably zero curiosity about our plans for the evening. In the end, I just had to stand up at the table to break the flow which sounds really mean but I had laid on a good home made tea and if I had wanted them to come to dinner, I would have invited them on a Saturday not a Sunday!
It’s a shame when people do this because you are reluctant then to invite them again even if you enjoy their company. Far better to keep things short and sweet and leave people wanting more!