Take him home. It's only 20 minutes - he won't die!
If you don't, you're being a bit martyrish about it. When they come home, will you just fume silently and seethe resentment? Or will you actually say something to them?
If you take him home with you, and make them come and pick him up, then you are asserting yourself - but not in a way that they can fault.
You need to make it abundantly clear that this is unacceptable and that they can't do this to you.