My two brothers both sit on the other side of the fence, jollyfolly, and from what I've seen happen to them and from what you say in your post I am sorry but YABU.
One of my brothers had to go to court to get perfectly reasonable access, and the courts DID tell the mother she was completely wrong - the benefits of a child maintaining contact with their fathers is so important and courts will not let practical issues stand in the way. The court thing was horrible for my brother and his EXP, as was all the stuff surrounding it - my brother had to get the police involved at one stage. It is now 5 years on and the relationship between the two parents is still poisonous, all as a result of the access disagreements (things were quite civil before then). It is not a very nice example of adult behaviour for my niece to have to see. This could happen to you.
The arguments you have are genuine concerns to you, but how would you feel in 10 years time if your son turned round and blamed YOU for not having a good relationship with his father? it could happen
You need to look at your arguments objectively if you can. Your child is 2.5 years, so 'Nappy rash' is going to look silly as an excuse in a few months time. There are plenty of people that feed their kids in Macdonalds and there are plenty of dads that never bother to make their kids brush their teeth too. None of what you say makes me think your Ex is going to be a bad parent. I would for sure be spitting feathers if I were in your position and someone looking after my son fed them crap etc, but he is your son's dad, warts and all. You cannot trade him in for a more acceptable version, no matter how much you might want to.
The sex offender thing I won't comment on as there is too little information. you feel you have grounds for serious concern here; we don't know why your Ex doesn;t see it as an issue. Anyway, if there is sufficient background to generate a legitimate concern in this regard then no solicitor in the land will advise your husband to fight restrictions if you choose to impose them.
The switches in routine and differences in discipline will without a shadow of a doubt make your life much more difficult. But that's just another sacrifice that needs ot be made for your son. You need to step away from what is best for you, and think of what is best for your child.
The more reasonable you are about all of this (and your Ex WILL get the custody he requests if it went to court), and the more civil the relationship you maintain, the better it will be not just for your child but also for YOU. You will be able to negotiate a proper 'time' for your DS to be returned on the Sunday; you will be able to set out clear rules regarding any other points relating to contact and you will avoid all the stresses of a court case.
In a perfect world it would be great if your Ex lived nearer, had a healthier diet and did things the way you would do them. but this isn't the perfect world, it's the real one. He has as much right to parent his child as you do, no matter how ugly that thought feels. Life will throw curveballs at you all the time: look on this access negotiation as just one of the steps of mothering that you need to go through; horrible, but it will pass. Make sure you have a reasonable solicitor to get things agreed, and try hard to move on. It needs to be dealt with and no amount of trying to make it go away will help matters.
good luck.