Can you override your instincts and give it a different perspective? If there are no other parenting concerns, and there's a good relationship between DS and his father, mid-week contact could be great for all concerned.
Your DS would then see his father on a 'normal' schoolday, not just for fun times, which means they have the chance to establish a relationship on a different level too.
Your ex would be more connected to the realities school life (I know, you may not see that as a bonus from your pov at the moment!) but it can be really valuable. He may also understand the realities of childcare on a weekday too, which can be helpful.
From your DS' perspective, he will always be 'going between 2 houses', whatever the contact ratio - to know that both houses are your 'homes' during the week, rather than one home and one 'visiting place' works fine, from our experience.
It's a long time from 5 through the school years, so don't forget this also would mean that you too could benefit from one day in the week where you're free from having to do the pick up/supervise homework/shuttle to after school activities!
It's a win-win all round. (Believe me, from experience, I know that it probably doesn't feel it right now, and you may be worried about 'losing' your DS somehow - try to envisage yourself ten years down the line and how proud you'll be of the shared parenting style you've enabled to support your DS to have the best relationship with both parents that he can!)