Well, you said he IS miserable, as opposed to acting miserable. I know some people find that distinction important when dealing with poorly behaved children as well. "he is being a naughty boy" as opposed to "he is a naughty boy". One is insulting the behaviour (which is fine), the other is insulting the boy (not fine).
He is right that if you're constantly calling him names, as opposed to challenging his behaviour, could be perceived as bullying. That doesn't mean you're a bully though, especially if his behaviour is bullying and that's what you're reacting to. Calling you a bully and passive-aggressive though could also bullying.
"shall we make dinner or do some chores" could be called passive aggressive because rather than asking him, you're voicing it in a way that sounds like a question but isn't really, and, could be perceived as pointing out he hasn't made dinner or done chores.
This doesn't mean you ARE a passive-aggressive bully, it could just be the way you have learnt to deal with his general laziness, and frustration at his failure to help. If you're constantly having to ask it gets extremely old and you get frustrated at having to harp on about things rather than him just noticing it needs to be done and offering to help.
Only you know why you act how you act, and only you know whether you're okay with how he's acting. You acting like a bully, or acting passive-aggressive doesn't get him off scot-free, it's not a catch all excuse for him not doing the things that need to be done. So my response to his "bully" and "passive-aggressive" comments would be "okay, but you still haven't done X or Y and you calling ME names doesn't change that".