The debt after 6 years ... I'm not sure it would be beyond the statute of limitations. What does the letter your mum have you set out regarding the term of the debt and repayment? If it says the repayment date was more than 6 years ago and she chose never to action insisting/enforcing repayment then arguably the debt can't be pursued. (The 6 year limitation period runs from the date the contract was breached - so, not when the loan was entered into but when you failed to meet the repayment date. Otherwise everyone could just stop paying their mortgages after 6 years, I wish!) If there was no repayment date set out (and/or the letter doesn't provide that any unpaid element comes from the estate) then legally you will probably owe nothing.
Morally - that's a different question. On one hand you have been enriched by borrowing money you did not repay. You admit that you have been able to repay the money for the last 5 years but have not done so. On the other you may be considerably worse off than your brothers and they are being "tight" not to let it go and risking a family fall out over a relatively small proportion of the estate.
My husband was in this situation. He had fairly wealthy parents and was one of four siblings who had had mixed success (or not) in life. A couple of the siblings were given a LOT of help during PIL's lifetime. I'm talking buying them houses/pensions/ paying grandchildren school fees etc. There was always mumbling about how it would be evened out in their estate and those siblings were just getting some of their inheritance early.
But then MIL died suddenly and FIL got dementia (then died a few years later) so the wills never got updated and the siblings were left an equal split. (Sadly one of the feckless siblings ended up predeceasing PIL). Youngest BIL banged on about how the large amounts SIL had received should be deducted from her share but it all got very upsetting. By then SIL had separated from her partner, was in a wheelchair and disabled. As a lot of the money had been paid within 7 years of PIL dying it did not escape IHT and to not only deduct the sums from her share but also the IHT would have left her with a tiny further inheritance for which she was in greatest need out of all the siblings.
Husband persuaded BIL that equal wasn't necessarily fair and had PIL been around to have a say they wouldn't have left SIL struggling so an equal split it was, with the IHT deducted equally from each share. BIL did gripe about it a bit but life went on and no one fell out. That's just an example!
I hope that you and your brothers reach a solution but I suggest you have a think about what is morally fair, bearing in mind that you had the means to repay and didn't. (Would it have left you consideably worse off than your brothers for example? What would your mother have wanted/done?)
PP are right though. You only owe £4k to the estate so don't get talked into parting with more than that!
Disclaimer: I'm a solicitor but not a probate one.