Based on the information you have shared, I would lean towards your partner's position that maybe you're not a safe partnership. Whilst on the one hand you can say it's not a regular occurrence so maybe it's not so bad, the knock on effect of irregular potentially violent flareups is that a child loses their stability/security because they never know when it might happen. It's always lingering in the background even if it's only in the foreground occasionally. And that's a different beast when you have no power to extricate yourself from the situation.
I also agree with your comment about violence being more of a values issue rather than an anger issue.
Your apparent suggestion that you effectively drive him to it makes me uncomfortable. If he wasn't violent and didn't feel entitled on some level to use violence then there wouldn't be anything you could do to drive him to it (genuine self defence excluded).
This stuff isn't black and white. We only have limited information to comment on, so observations, weightings, interpretations are going to be variable. Although frankly you often get the same mismatch in verbal communication.
You have quite an aggressive posting style. I am a bit baffled as to why you think you'd be able to create constructive dialogue, or encourage people to be helpful or patient, from such an aggressive position, launching into diatribes and then telling people they're not allowed to comment on what you've dredged up.
If you think your aggressiveness (because it's certainly not assertiveness) is contributing to volatility in your life (by which I mean avoidable conflict being created, not violent reactions) then maybe it's worth working on. However, the way he responds to that from you is his issue and I expect, as you observed, would be the same with any woman who was "challenging" him. Arguments don't need to escalate and that's not a pleasant environment, but him "losing his temper" shouldn't result in violence.
And no, I don't consider myself perfect by any stretch of the imagination.