To wonder the thought of my ex makes me feel like this

[AnotherSharon]

Honestly, the thought of him coming to pick up the kids and for me to see him makes my stomach turn. Anxiety is through the roof! I feel sick at the thought of it and when I do see him my body seems to shake?

Arrange pick ups at another location, somewhere you can drop the kids off and he picks them up (so your don't meet). If this can't be a friend/family, there will be some service that offers; think DV type scenarios, enquire at local council.

Why do I feel like this though? He wasn’t violent but very manipulative and he puts zero effort in with the kids, it’s such a horrible feeling

Aww, you poor thing.
How old are the children? I'm guessing too young just to usher out of the house towards his car?!

I think it's perfectly natural for a loving mother to respond that way to having to give her children over to someone like that, even if he is their father

Your body is reacting to the stress he put you through previously. Like it's getting ready for battle. Don't let him indoors, smile and wave the kids off, same for when he drops them off.
(((hugs)))

They are 4 and 7.

Completely understand this feeling, I get it too. It is your body going in to fight or flight mode because you see him as a threat.
I would definitely suggest doing handover in a public place, mine is at a Tesco with CCTV. It gives me peace of mind. We now don't go near each other, the kid's walk over. My body still goes into that heightened state but it is far more manageable knowing he isn't showing up to my house. Before I would be on edge for hours before pick up.
Really feel for you. If you haven't already, consider going to your GP and accessing therapy. It has been really useful in managing these reactions that I have and supported me with the children.

It sounds awful but I really wish he just wouldn’t be part of any of our lives anymore. I honestly thought he was such a good person in the beginning. I can’t believe what he has put us through.

I ended up bumping into him at the house we used to share and I was shaking so bad.

Doesn't matter why, deal with preventing the feeling first and you can always find the why after. It may be your just overwhelmed. Just because someone didn't batter you, as you've said he was manipulative, he sounds like a charmer, you've every right to not want to out yourself through being in touch with him. Your body knows he is one hairsbreath away from causing shit just because.

Just because...

Prevention is better than cure.

Find a service that enables drop offs and set it up. Then he has no control over your "peace"