Okay, I hate to say it, but I was this sort of child.
.
Anything that I saw as an invasion of my privacy, or demands that I clean up immediately, just made me more stubborn and resentful - the room got cleaned in the short-term, but I saw it as something forced on me rather than my responsibility.
What helped with me was having a bit of choice in things - almost like you'd do with a toddler, really. 'Do you want to tidy up now, or when we get back from swimming?' and so on. Also, help to get organised. I'd want to clean up my own personal papers and other private things, but at 11, faced with a floor full of mess, the task would seem too daunting to begin. Maybe tidying every day for a set amount of time (15 minutes before bed?) would seem more manageable. This is what I did when I was severely depressed as an adult - I'd literally clean for 30 minutes, and then stop, even if the room I was trying to clean wasn't finished. A few weeks later, I had a tidy house, and it never felt overwhelming.
I think if punishing him for being untidy hasn't worked before, and he's in any way spectrum-ish (like me!) then it's not going to have the desired effect. Instead, think of ways you could get his co-operation to develop better habits, so it seems like you're working together on keeping the room tidy, rather than 'you against him' which is how I perceived this sort of thing as a child. If you do move him to a smaller room, frame it as being easier to keep clean/organised, rather than as a punishment. I know I'd have been thinking 'what will they do if I don't keep the small room tidy, put me in the garden?' and it would have thrown up issues of spite and resistance. Think about the long-term outcome you want - which I know is hard, since his behaviour must be so annoying for you!