oh yes, you are sorting too much. FAR too much.
I just do whites (every now and then when they are looking mucky) and non-whites or if in a rush, shove it all together and put a scoop of vanish in.
Only other separating I sometimes do is a separate wash for all the towels (because they are so big and bulky and take up washing space).