I know exactly what you mean cupcake 
You did a brave thing the other day in getting him to go, could you possibly sit him down and say your not happy with how he handled that and maybe give him until the next week off to see how you feel and if you are doing the right thing?
Your happiness is very important! 