Fucks sake....PLEASE PLEASE get Social Services involved. And considering that you never ever "have" to utter the words "you are not seeing your children". That's his opportunity to mess up.
Just read this thread in its entirety and the OP isn't clear once at all about why there's all this texting and doorstepping and shit, just that "I don't want him to throw me not allowing him to see the kids back in my face".
My darling OP, it is evident you still hold out hope he will come to his senses and your family unit restores. You are vulnerable, you have just had a new baby, his parents are arseholes....but you have to drop this dream as it won't happen. EVER. He's showing you time and again.
So contact SS. They are there for you as he wont be. Your children deserve a happy mum and a stable environment. Persisting with dealing with it solo is not working and your kids will suffer, I'm sorry.
It's time well overdue for you to look out for you and them and use official channels only.
No court in the land would mistake your actions as withholding access, but I don't think your issue is truly about that at heart.
You have a dream that sadly won't come true. Time to aim for a new dream life where you and your two kids are happy and independent and loving it.
Sorry for any harshness but you need to wake up and start manning (womaning) up ASAP.
Tomorrow is the first day of the rest of your life 