my ex did the same thing...I was always the one at fault, told me I was going insane and needed to see a shrink (he has serious mental health issues himself and was projecting), whenever I confronted him with reasons I was upset / needed help / needed his behaviour to change was told I was oversensitive etc, when in fact he was being an arsehole. got to the point where I nearly had a breakdown, broke up with him and life improved massively (we had a DS together so still implicated in my life but thank god not together any more)
some classic passive aggressive (and generally abusive) tactics are:
constantly late (sometimes excessively)
says one thing and does another
lies
manipulates
withdrawing / sulking
taking out their anger in other ways (won't confront it at the time you bring it up, won't discuss, then later gets really angry over something stupid, blows up)
breaking your stuff behind your back so you can't prove it
the silent treatment
gaslighting (making you think you're the crazy one)
never confronting issues, turning them around at you
undermining you subtly so you can't put your finger on it
you feel like you're treading on eggshells, changeable moods
etc etc ad infinitum
There's little you can do to change it and the best thing is get out ASAP as you will be in a downward spiral otherwise. Find some strong people (friends and family) to help you out of your current situation and get out as quickly as poss.