I think leaving the ECHR is the end game for lots of Tories, not immigration at all.
At the moment the ECHR, HRA and EA together are viewed by some as "red tape" that stops governments/employers from doing all kinds of discriminatory and prejudicial things and thereby impact their ability to be as capitalist as possible (e.g. by having to make reasonable adjustments for disabled people, or provide paid maternity leave).
Therefore the same people who wanted brexit also want this.
I also think this is what's driving the "sex based rights" malarkey - its a trojan horse to scrap the EA. And that's why the Conservatives haven't made any progress. If they wanted to strengthen sex based rights that would involve a wording change and there is cross party support so it should be straightforward. But they don't want to do that. Badenoch wants to scrap it altogether and replace with something else. And I don't think that's to protect women.
(Sorry, slight derail. As a feminist all this talk of scrapping the HRA/EA, coming out of ECHR terrifies me)