Im so sorry to say this, but you are aware that there are some, very strong alarm bells ringing here from your post don't you?
Firstly, I would be saying that a man who has voiced an opinion of your children being baggage, would make me think he genuinely has some belief that your children are, in some way not a completely positive thing. That coupled with him not wanting anything to be touched by your kids, I would come to the belief that he doesn't really like kids, or atleast your kids. I'd not want him to build a relationship with them any further because it seems he is already struggling somewhat with them.
Another biggie is, he has threatened violent and destructive behaviour towards you. If he is threatening it this soon into your relationship, what will he be doing in a years, 2 years, 3 years time? What with him seeing your children as baggage, I'd be concerned they might be in the firing line of his temper.
It's obviously up to you, but you need to take into account he's threatened to smash your door down, to what end? Surely if he smashed the door down, he wouldn't then walk away would he?
It sounds dramatic, I know. But I've been in this situation. it spirals. It starts with hitting the doors or walls. They then throw stuff in your direction, telling you they'll never hurt you. Of course it wasn't deliberate that things either did or nearly hit you!
They'll then hit you. Even that will get worse as time goes on. You may even find yourself trying to explain "accidents" to your children.
Please don't ignore these signs. He needs anger management counselling. If you are going to continue with this relationship you need to step back until that's done.