.he won't take them to his home, he refuses to tell me where he's living even, all he said was he has a flat and it's none of my business.
I believe that by law he has to tell you where he is living if he is expecting to have the children - as their mother you have a legal and moral right to know exactly where your kids are. The courts would certainly enforce this!
As others have said, you are still letting this knob emotionally and verbally abuse you and you need to stop it.
I believe i have read someone above suggest you right out a contact schedule - this is an excellent idea. You tell him that you will make contact visits available on x,y,z dates. If he needs to cancel or amend the dates he needs to give you a minimum of 48 hours, he turns up at time A and returns them promptly at time B. Throw back at him the crap he is spouting!! HE collects and drops off - if he is soooooo concerned about you being frivolous with money then why does he want you doing 4 journeys for a contact visit?!
You tell him, via a solicitor if necessary, that ALL contact is to be written via email or text, you will not discuss on the phone if he is so concerned with your tone of voice - that way it is matter of fact with no risk of emotion.
And if he continues to abuse you or cancel without warning you will withdraw contact and see him in court. Your kids come first and he is doing them more harm than good by messing them about!
If he see's you have the strength to tell him to piss off and mean it he will hopefully start acting like a human being...although i wouldn't recommend holding your breath 