I really don't want to attract even more contradiction to your thread, soggy, but I do want to validate your unease.
You wrote: I do something a couple of times and then it is like that is it and he will make changes to plans, etc based on this assumption that I now "always/never" do such and such
There is a sort of dysfunctional thinking that therapists call "eternity thinking". People who suffer from it do, as you describe, assume everything will always be the way it is today. It's incredibly limiting. I briefly went out with a bloke like this - he'd order my food for me (red rag to a bull, that!) based on what I'd said I liked last time we went out. We even had exactly the same sex each time. When I changed things, he'd go "Oh, so THAT'S what you like now?" as if I was wildly unpredictable and/or doing it to annoy him 
I have wondered if he had Asperger's but, no. I think he was just weird.
One consequence of being with someone like this is that you have to go through this ridiculous rigmarole every single bloody time you want to just do what you do. Hang your coat up differently: "Why have you hung it that way? You always do it this way!" - and he moves the coat. Make tea instead of coffee: "But you always have coffee in the morning!" No, not always. Today I'm having tea, look. "Oh, so now you like TEA in the morning?"
Gah.
CBT therapists love working with this kind of dysfunction. But first you have to get the person to the therapist, which means getting them to understand they're being weird. Not easy :(
I don't like this at all: he lists various occasions, often jokingly in a sort of putting his arm around me and saying "come on you know you're useless at organisation" etc
Telling you how crap you are IS abusive, especially when it's done all the time. As Hissy says, he's defining you and your reality. That really isn't on. Without knowing more, I can't guess at whether he's on a mission to undermine you or if he genuinely believes you're unreliable because, to his strange mind, you're unpredictable. Either way, it's a pig of a thing to live with. Deliberate or not, it does undermine you.
Have you tried just agreeing with him, then going on your merry way? Could you even do it, or do his constant rearrangements of your life make it impossible to ignore?
Has it been getting worse? I'm a bit worried that you may end up doing everything to an exact routine, like a little clockwork toy, just for a quiet life ...