YANBU. My DH used to do this. (Some years ago when the kids were primary age.) I would get up on a Saturday morning (sometimes Sunday,) and he and the car were both gone. He NEVER left a note, and he had a crappy PAYG mobile phone but NEVER took it, so he was uncontactable.
It pissed me off because
a) I may have wanted to go out myself, on my own or with the kids.
And
b) I couldn't settle with a TV show or film or anything, because I didn't know if he would be back within half an hour!
It used to feel a bit like a manipulative control thing. He may not have meant it like this, but it felt like it. He fucked off with the car with NO PHONE so he wasn't contactable, and I was stuck in, not knowing when he would be back or where he was. He would come back anywhere between an hour and three hours after I got up. 'Just been to my mates/my cousin's/a walk by the river, the shopping centre etc....'
He did it maybe 8 times in about 12 months this one time. So I finally did it to him. Got up at 7am, took the car, left the kids and him in bed, left my phone at home, and fucked off out at 8.15am til about midday. He was STEAMING when I got back. Where the fuck have YOU been? I have been worried out of my mind here, and I wanted to go to the shopping centre this morning. FFS! You coulda left a note or taken your phone!' 
I said 'YOU never fucking do!' Then I pointed out how many times he had fucked off out and not left a note, not said the previous night that he was going out, and had not taken his phone!
He grunted, huffed and puffed, stormed out of the room, and always took his phone after that! Although he did it a lot less after that too!
I think the poster is correct who said they don't/won't say anything, and tell you where they are going, and how long they will be, because they are DA MAN, and it feels (to them) like they're asking permission.