OP
Personally, I would be very wary. I know you've said what he has told you checks out- but have you had a thorough look into it, or just checked the conviction/time served etc is right?
I agree with a PP- manslaughter is death caused by reckless or negligent actions that a reasonable person would recognise as posing an appreciable risk of causing death. I don't think killing someone in self-defence would lead to a manslaughter conviction. Almost certainly not one where the sentence is 5 years.
TBH, I think your dad is right to be concerned, although hasn't got the right to tell you how to live your life.
From later posts it seems you have been in at least one abusive relationship- you feel this means you will recognise warning signs. Given that circumstance, it's even easier to understand why your dad is worried for you- and your children. Even if they haven't met him yet, presumably they will at some point? And if anything happens to you, they will be affected. So your children are involved, even if only peripherally for now.
In my experience many people think they will "notice the warning signs" but they don't until it's too late. Be very sure of your ground, take things very slowly. If he is the good man you think he is, then he will understand why.
I do think, though, that you minimising what happened somewhat- again not unusual to want to defend/minimise when you care for the person. But you need to look at this dispassionately. This is a man you may want your children to develop a relationship with if things become more serious.