Those MPs that have quit labour and Conservatives were voted for by their constituents based in them being a member of a certain party, they're not in the party anymore so they're no longer who the people voted for.
Does that make sense?
Shouldn't they be stepping down and letting their seat be voted for again?