Ahh, I think I would bin this one...
Exactly what folk say above about cumulative effect.
He accuses you of ignoring him if you miss his calls? Unfair. Nasty.
He likes to wake you up if he's up? Don't like that either - something very mean-minded as well as controlling about it.
I wouldn't have thought anything of the Brownies comment either! It was your childhood, you're lucky to have had a happy one.
The thing with relationships which turn bad is - usually, when the little alarm bells start ringing, they seem such trivial things. So you ignore them. You ignore them during the easy to get out phase, when actually you've invested very little, and it would be easy to go. The irony... because by the time the red flags are quite clear, usually when a nasty piece of work relaxes the good guy act and shows his true colours... well, it's a year down the line, you don't want to throw it all away, you've got a holiday booked... Much more difficult. And you end up rationalising and staying...
So, LTB. Really. The signs are not good. Moreover, they're signs you've noticed all by yourself. They really are there. But you aren't going to get 'proof' of the fact he's a bad un for a while yet, probably. It's a leap of faith - faith in keeping standards high and not settling for something which makes you uneasy - but if you get out now, the fact is you're probably saving yourself a whole heap of shit.