I can see why you feel defensive. I'm sure you expected everyone to support you, say you should just go, and validate your choice.
But given that you asked for advice/opinions, and given that the majority of the responses are that perhaps this is not fair on the kids...do you think you should just pause for a minute, forget about being defensive, and think about this for a minute.
Is it possible that your desire to move away, move in with your new man, start a new life (all completely understandable) and your dislike of your ex have coloured your judgement? Lots of people have suggested compromise positions, which you seem to have ignored. Are you secretly looking for validation of your choice because you know it isn't really fair?
I'm sure your ex is far, far from perfect. BUT - my SIL smokes dope, regularly - she is a wonderful mother. My DSS's mother, should I list all the negative things she has done, would sound terrible - but she is my DSS's mum, he loves her, and whilst her way is not my way, she's a bloody good mother and he is a fantastic kid.
You sound like a smart, ambitious woman who wants to do the best she can for her kids and there is no way you would let your kids have EOW + midweek if your kids were in serious danger from a drug-addled lunatic.
Can you think of any level of compromise that would support your children's relationship with their Dad?