Change the locks and don't give him a key. If he has the children, why would he need to take them home? If your neighbour was taking the children out, would you give them a key? No. If they needed anything, it could probably be provided by taking them to his home or it could wait until you are back.
It is only legal to break into a house if it is your primary residence. E.g. if you go on holiday and squatters move in changing the locks, you have every right to break back in. If, however, you have a holiday home and squatters move in, you cannot break in. You'd have to get an eviction order. So, as it is not his primary residence, he would have no right to break in if you changed the locks. If he wanted to move back in, he would have to apply to the courts and then you could raise the question of his violence. He would have to show a very good reason why he should be allowed back, when he seems to have been coping quite well elsewhere for the last 4 months.
CHANGE THE LOCKS.
Oh, and the tax bill is his.