@woopsididitagain I really feel for you as you are currently where I was several years ago. It's only been during the past 18 months, I would say, that I've managed to stop. I really recognise the soreness you describe, too. It's truly awful. Not to mention the constantly hiding my hands so people wouldn't see the state of my fingers.
Like you, I tried everything. But what has eventually proven successful for me (may not for you but I offer it from my personal experience) is meticulously clipping all the small loose bits of nail as soon as I become aware of them, as otherwise I try to bite them off without even thinking about it. I bought this little kit from Amazon and really only use the clipper but sometimes also the file just to 'shape' hard skin around the nail. I also bought this cuticle oil and I believe this is what has really helped the most because not long after I started using it, I noticed the soreness easing and that made a huge difference for me. I just keep one bottle upstairs by my bed and one downstairs right next to where I sit, and apply just a tiny drop at the base of each fingernail throughout the day/when I go to bed (I do wear cotton gloves during the night). If I get up for the bathroom during the night, I remove the gloves and apply the oil again, and the gloves, before I get back into bed. It might sound like a faff but really it doesn't take more than a minute and the oil is absorbed quite quickly.
I'm happy to say that my nails have grown surprisingly quickly over the past four/six months in particular and are nice and shiny and ridge-free. I haven't let them get past the tips of my fingers but just gently keep the tips rounded and really keep on top of loose flaky bits as I find that those are the worst culprits for causing me to bite in the first place.
Other posters have mentioned the link to stress and anxiety, etc, and I recognised that this has been true for me most of my life as my childhood was quite abusive. You don't really think about these things being linked, but they are. As I've grown older and recognised the truth about my childhood and a misjudged, unhappy marriage, I've been able to identify ways to help me relax and take better care of myself. Currently working on my hair and skin :-)
I do appreciate that we want our nails to grow out overnight and look better and healthier. But I also appreciate that we do genuinely know this is impossible, and for people like you and me, it's just as much about dealing with the actual physical pain the biting causes, the worry over whether or not we've gone too far and cannot undo the damage done, and the fear of developing infections that might make things even worse, if that were possible.
I don't have a complicated regime. Just the clipping, the cuticle oil (that particular one, for me, because as I said I tried everything including dozens of other products all pomising miracle growth etc) and wearing the gloves overnight. But the combination has worked and continues to work so I really hope it will work for you.
Please return regularly to update your post, OP, as I would like to support you on your painful - but NOT hopeless - journey. I wish you well and send a virtual hug of encouragement. I'm sorry I can't post any pictures as I don't own a camera.