I think it is because your vote supports racist policies and reinforces inequalities. Your DP is therefore drawing inferences on your character.
It sounds like we are similar, except my children are in state school. I have, very few I should say, friends who vote tory who are black, were raised in social housing, attended the same state school as I did, windrush ancestors etc. I have had to battle and struggle to see past that. I now just ignore it.
For me it's a case of pulling up the drawbridge after you've crossed safely. The Conservatives have grossly underfunded state education in this country. I wonder whether you care about that, given you benefited from better funded education 30 years ago. Your black family might have come over legally when Britain was on it's knees and begged them, but do you also know of more recent people forced to 'work' the system? The countries they've come from are still bent double under the weight of historical loans and debt linked directly to colonialism. I wonder why your sympathies are with your friends who might have to swap their children to state school and not the 2024 version of you as a child who doesn't have paper to write on in said state school. You seem to just care about yourself now. What about you 30 years ago? What about that person, today?
The PM might be brown/black but he largely denies racism, as do all of his fellow cabinet members, I believe. They stand by Frank Hester. Your vote supports that heinous, and very recent, behaviour. Are your children visibly brown skinned? If so, think beyond private school years to them driving while black, working while black, walking down the street while black. The Conservative party has a terrible record on race equality policies. Just look at the civil service where they are removing all DEI posts.
If my partner voted Conservative it would impact our relationship as it would indicate our values had hugely diverged. I wonder how this will sit with your partner over time.
You're possibly a great, loving person, but for me, you do sound quite socio-politically naive or self-interested (as your partner said) and, I think, myopic.
But good luck. As Lauren Hill said, we have many paths and we must choose one. You might have some wins with a tory government, but you and certainly other ethnic minorities, will have some heavy losses. I personally can't do it.