Weeeeeeeeeeel, to be honest, YA both BU.
He shouldn't have called you at that time of day, especially to ask you to come in early. Anyone with sense knows not to call people with children between 6:30 and 8:00pm as it's the crazy time of the day.
However, you should calmly have asked him to call back later as you were in the middle of something, and then hung up on him. It's very, very rarely appropriate to yell at a colleague, no matter how much of a numpty they're being (and I speak from bitter experience )