Ahh -I think you've hit on something there.
Routine can be HUGELY important for some children. My ds was one - he would get VERY fractious if we had a 'free and easy' day.
This is what our routine was at your DS age. FWIW!
He woke up early so we were up and having breakfast by 6.30am/7am at the latest.
He sat in his booster seat for breakfast in the kitchen and I sat with him (no telly at this point)
Then after breakfast, I got him dressed and we played in the living room, including about half an hour of telly (cbeebies!)
Each and every morning we went SOMEWHERE, whether that was playgroup, the park, swimming, whatever.
After that, home for a nap. It was at this age that he started to not want this so much so I gave him some quiet time in his room. When he started hating that I just allowed him to nap in the car on the way home from wherever - I drove around a bit more.
Afternoon he had a snack and more playtime and he was usually ok because he'd been out and wasn't too bored. Often he had a bath in the afternoon as he loved 'playbaths'
Dinner time was about 5pm again in his booster away from the telly.
Straight after dinner it was into pyjamas, then milk and a story and bed.
I am not saying to follow that BTW just to show that we were very routine bound! However DS needed it and it worked and I believe made him quite a secure child - I do think many, many children benefit HUGELY from knowing 'what's coming next'.
I definitely think you might have found the 'key' to your ds there......