Oh sweetheart, your little one will be fine.
It's not ideal at his age, but it won't be harmful either.
DS will probably display a day or two of re-adjustment behaviour after each visit, which is hard for you. But if you can be patient each time whilst DS re-adjusts (to his home, his bed, his routines), then he'll do just fine.
Either your selfish Ex will soon tire of doing the travel and childcare and drift away, as perfectstorm suggests.
Less likely, this could be the making of EX - he steps up, learns to care for his son, and becomes a real part-time dad to DS. It sometimes happens I wouldn't hold my breath.
Does EXH have a mum, or gf, who will look after DS for him? Brace yourself in case he's that kind of hypocrite. Whilst it's frustrating, remember that if it's the same consistent person each visit, then DS will be fine, even at age 2.
Disagreeing with many posters above - I wouldn't big up the visits to DS. He might feel like you're sending him away. I'd take a matter-of-fact, teacher-like approach.
Don't tell DS it will be great, don't tell him it will be awful - it just is. Leave space for him to have his own reactions. With luck, he'll soon just accept these visits as a fact of life.
Finally, plan fun things for yourself. Go out, to the pub, or movies, see friends. Tackle chores you usually can't get done with DS around. The time will go much faster - and realistically, take advantage whilst you can, as odds are EXH will soon lose interest.
Good luck!