It's interesting: I think Left and Right have such different value systems that they both genuinely see the other side as racist and also genuinely don't consider themselves to be.
Values
Labour are driven by their hierarchy of privilege. Their value system favours a high level of state control over individuals, as a way of equalising outcomes, with no difference due to either privilege or due to individual merit (in fact, they seem to see individual merit as just another form of privilege to be wiped out).
Conservatives are pragmatic and favour successful outcomes over uniformity/equality of outcomes. They consider success - and it's rewards - to be individual, and are opposed to the state levelling down.
Outcomes for Racism
Labour see race (and also being an Asylum Seeker) as a cause of reduced privilege, and see it as self-evident that this should be balanced with preferential treatment - that's their 'neutral' point.
So they think Conservatives are racist because Conservatives object to the preferential treatment Labour see as neutral. But the Conservatives position is precisely because they aren't racist: and consider each person to be an individual responsible for themselves, and race is irrelevant to that.
Labour assume they aren't racist themselves, because with their automatic backing of the group of least privilege, they often do back non-white groups (eg AS, Palestinians, the US black civil rights movement in the 50s).
What they don't have the self-awareness to realise is that their MO is actually based on simply opposing anyone they percieve as having privilege. Very reductive, but self-righteous anger is very gratifying.
So sometimes their backing is reasonable (like the black civil rights movement of the 50s). But other times their 'Right Thinking' lacks any kind of nuance (hence an inability to balance rights between white working class and AS, or between Trans rights and women's rights). And they tend to rely on there being a bogeyman (hence weekly demonstrations for Palestinians and nothing about the genocide ongoing in Sudan). The right see this broad-brush categorisation of who to oppose and who to support - lacking context or nuance - as actually being racist.
Labour values also prioritise state control, so anyone who they think should be dependent on them - but is successful enough not to be - is a threat. Hence the persistent anti-semitism, and probably also the particularly virulent attacks on Sunak and his wife. The Right (correctly) see that as Labour being racist. The Right think anyone is equally entitled to create their own success, and race is irrelevant. They hope that a rising tide lifts all ships.
That's why the Conservatives have had many female and non-white leaders (because they value capable people regardless of race and sex) and Labour haven't (because they see people outside their expected pecking order as a threat).