I tell my ds1 that he can go and sit by the door until I am ready to leave. He soon gets bored and goes back to play. After a good old cry that is. Depends whose house it is, if there's a reason he doesn't like it or if he;s just being a pain as well.
I think I'd be firm and say no - we'll go when I'm ready (am guessing you've also gone to have a chat with friend etc). Then not leave it too long. Otherwise, where does it end? Remember my 3 yr old being quite manipulative in this way. Not quite sure what age your ds is though. Certainly at age 4.5 onwards I think they can understand it's rude and need to know it's not on.