I agree with RandomMess - you should no longer provide YOUR home for access, and you should pass it over to Ex to see what he wants to do (believe me it will come across better all round).
I had a similar-ish situation not that long ago. My Ex thought he should be able to visit/ take DS whenever he wanted - never mind his routine, his bedtime, or the fact that I worked. He wanted me to be able to make DS available as and when, at random times, often with very late notice, wanted to do it in my home etc. This worked for a while when I was on maternity leave, but when I went back i just couldn't do it anymore.
I went round and round for months trying to come to a solution, trying to find ways that could work, ways for DS to see his dad and for my Ex to be happy with. I even tried on occassion to convince Ex to try certain things that would make it simpler (like speak to work about his shifts).
What I eventually realised is that it is Ex's responsibility to figure all of that out, as long as I wasn't putting any unreasonable barriers in the way. I could come up with a thousand solutions, but the majority entailed him doing something. And unless he was willing to do it, it wouldn't work.
For a while he wasn't willing to make any changes, and we rowed, a lot. I even went to a mediation place but he refused.
Eventually I put my foot down. Told him I wasn't going to come up with solutions, or bend over backwards for him to throw it in my face. I told him I had a job to do, which was providing 90% of what my DS needed, and therefore could not be fucked about. I told him when DS would and wouldn't be available and that I would need notice to make any changes to this. From that moment on I drew a line in the sand and stuck to it. I even wrote it down and just kept reffering back to it.
It has settled now into an every other weekend thing. I would like it to be more, as I believe it's a bit long between visits for DS, but Ex works shifts, and is not inclined to speak to work about regularising his days off.
At the end of the day it's his choice, and he needs to work it out for himself. And you are right not to want him in your home.
At 2, she can pretty much be taken anywhere, the park, for tea, whatever. So hi excuses are pretty lame.