Tempo , sorry to say this but i dont think you have a problem with communication at all , i think you have a problem with a controlling emotionally abusive man.
Does he walk away when his boss is talking or one of his workmates ? Of course not , he saves it just for you , in private , where no one can see . You mention feeling shaky , unsure of yourself , paranoid ect .
These are all side effects of living in this sort of atmosphere , and its designed to keep you on edge , to keep you checking in on him , and it works doesnt it. Have a read of some of the older threads about emotional abuse , and search for stonewalling ect .What hes doing is typical , so typical in fact there are many books written about this sort of behaviour.
Your financial arrangement is very unfair , the fact he says he will " allow " you access reflects his attitude about finances .
Are you worried that if you dont make an apropriate amount of fuss for his birthday he,ll later put you on a guilt trip because you didnt make enough effort ?
For now , id stop trying to communicate with him , fuck him , he,ll be revelling in it , make no mistake . If you do stop trying to communicate with him fully expect him to turn on the nice guy act , in fact he might even start a few discussions about this himself , to draw you back in.
Start educating yourself about emotional abuse and the tactics hes using. Once you can see what hes doing and that its not about you things will look very differant.
Dont waste your weekends tiptoeing around him seeing what he wants to do , make plans to go out and see your freinds , and yes he,ll probably pull his face , but he will anyway , you just cant win.