"...if there's a right to access your house then anyone can access that."
Generally speaking, yes.
"If they then further access something behind your house using just your land then that's not a violation of any access rights."
It most certainly can be in violation of any access rights, it depends entirely on how the deed is written.
OP, you really do need to read what the deed actually says. Without that, nobody can really help you or give any sort of meaningful advice.
I would suggest that you also post this on the gardenlaw.co.uk website which is a forum that deals with legal issues surrounding rights of way, boundaries, hedges etc. Although, even there, the first thing they will ask you is to give the details of exactly what is written.
Depending on what it says, you may well be able to build a new home in the garden and have access or you may not. No one can say without knowing what the deed actually says.
I would suggest that you start by taking legal advice from a solicitor that specialises in this area (just google something like "rights of way solicitor") who will be able to look at all the documentation and give you a realistic appraisal of your chances.