The HRA implemented the European Convention on Human Rights into Uk law. The Uk was already a signatory to the Convention and was bound by it, so the HRA just means that UK courts, rather than the Eur Court of Human Rights in Strasbourg can hear human rights claims.
So even if the HRA was repealed, the uk would still be bound by the Convention. The organisation responsible for the Convention is the Council of Europe which is nothing to do with the EU. the Council predates the Eu and the UK was a member well before it joined the Eu.
So human rights "problems" shouldn't be blamed on the EU.
Also, the Tories aren't proposing withdrawing from the Council of Europe, or not being bound by the Convention, so we'd still have the same rights.
And of course (bigger picture), these rights are very important and valuable. It's why rape victims can't be cross-examined at trial, it's why we can peacefully protest, it's why states can't engage in degrading and inhumane treatment. How can the right to life, to liberty, to freedom of assembly and expression not be something we cherish and defend with all our might?
And as for why criminals have rights too under the HRA: well the clue is in the name.