I agree it's rotten manners not to phone you, in fact he should have asked if you minded.
I don't mind my DH having time out, and he's great when I want some, too. But we always ask each other beforehand so it's all arranged. He's taking you for granted and treating you like someone who doesn't matter. He's lucky you're cool about his taking time out, some wives wouldn't accept it at all.