Ok, this is all still very new for you. It's raw, and may be a lot of people in RL & online telling you that you have to bend over backwards to facilitate access - that you are responsible for making it work.
^ this isn't correct. Yes you shouldn't block access, but no way do you have to have it at your home. It's his access & time to spend with the children, not time for him to be with you. He will have to find a suitable safe location in which to have the children. Either at his parents or pick them up and take them to a park or at worst - if he really doesn't have anywhere, a contact centre.
The arrangements need to be written down, find a family solicitor & get an outline of your rights. Some will give you 30 mins free & charge you a fee for writing "your position into a letter/statement for you". Knowing your rights can save you a hell of lot of grief going forward.
Set dates and times for access, be reasonable. Set it out in writing (you don't need a solicitor for this). If he doesn't agree, then mediation will be required. Btw you don't have to attend mediation with him, it can be separate - if you feel uncomfortable around him, or if he is aggressive - I would suggest telling the mediator this prior to booking an appointment.
If he refuses mediation, and you have offered sensible access (not at your home - as this is not reasonable of him & will not be considered reasonable) - he won't be able to proceed with anything.
I suppose what I am trying to say, is you need to take back control & order for your children's and your sake. And get all your ducks in a row. It will make your lives so much easier.
Ps I tried the nice nice approach, always backfired, Home visits just got worse and worse - it didn't benefit the children at all. Once I had everything laid out & set, yes he didn't bother with them for 6 months, but the last few years haven't been too bad. But I always have the backup of knowing my exact legal position for when he does screw up again.