He speaks to me and his dad like crap
No neurodiverse or any other condition makes this acceptable. Social communication can be a huge area , but rudeness should not be tolerated.
Tell him there needs to be changes in the way you all communicate with each other as a family. Start as though he was a toddler, explain how to ask for things and how conversation works, the turn-taking, the polite way to ask, using please and thank-you etc.
Remember, no matter how you model 'good behaviour' he won't absorb that by osmosis. Be aware at all times that he doesn't automatically understand the rules of social communication, he cannot see or understand inferences and hints so your communication to him needs to be very very straightforward.
Instead of saying 'tidy your room' or 'pack your schoolbag' then get frustrated when he appears to be clueless, break each task down to one simple instruction at a time, then when he's done that use another simple instruction and demonstration if necessary until the task is completed.
His academic intelligence does not mean he should instinctively know how to do other things. He needs guidance and instruction on some things. Look up Executive Function.
Perhaps research ASD and ADHD and learn about techniques you can use to start to ease his anxiety and need for rigid control.
www.autism.org.uk/
www.additudemag.com/category/parenting-adhd-kids/
Be aware that neurodiverse kids can have a large gap between their emotional age and their chronological age, the ballpark for that is usually around two thirds, so if you perceive him to be emotionally immature, that's okay, take that into account to manage your expectations of his emotional capabilities.
His routine is ridiculous, as soon as it’s 12 he comes down for lunch and he can’t wait any longer. He panics if it’s later
That's him trying to control his anxiety by imposing rigid rules, you need to help him to be more flexible. It's not an easy task, lots of time and patience required.
Start to vary your lunchtimes. If he needs a weeks notice to get to grips with the idea, do that and remind him every day that's what's going to happen. Then on the appointed day, tell him at breakfast, today, lunch will be a t 12.15. And stick to it.
Also look at Ross Greene's The Explosive Child and his website Lives in the Balance.
Once you understand why he behaves in certain ways and you have the knowledge of how to accommodate or circumvent that, life will be a lot easier, but it takes a lot of time. 