Yes, this will cause huge problems down the line potentially.
What would happen if your neighbour wanted to build an extension up to the boundary? They say the boundary is where the fence now is. You say the boundary is where it was.
What happens if you want to use the old boundary line for something? An extension, a shed, a gym? And the neighbours say no, the boundary has changed.
See my user name? Well, I am moving because of a fucking fence. A fence had been put in the 'wrong' place many years ago. We moved in 19 years ago, the fence was unbeknownst to us in the wrong place. 2 years ago a new neighbour moved next door. He wanted his 4 inch strip of land back. He didn't have a leg to stand on legally as the fence had been in position for over 40 years.
But it STILL cost us to fight it. In the end we had to give in because he wanted to go to court. Our solicitor said we would win BUT even if we were awarded costs we would still end up paying around £10k for the privilege of keeping our garden the way it was when we bought the house.
I was in anti depressants, it was horrendous. The neighbour was totally vile, laughed in my face, gleefully dug up my garden border to get his 4 inch strip of land. All my plants were killed.
I can't live beside this man, so we are moving, which of course has cost thousands.
My advice is offer to pay to replace their fence. It will then be yours, so they can't touch it, which they may not like! But the cost of taking the fence down is nothing compared to what could happen.
There are plenty of horror stories like mine if you search.