I'm like you and have one close friend but no groups or anything like that, I don't hide it.
I found that I was initiating contact ALL the time, when I realised this I thought "Am I more annoying than I thought? Am I more boring than I thought?" then realised I didn't care anyway, so I stopped initiating contact and I don't speak to anyone anymore. I hate going out, it induces really bad anxiety attacks, and not even having to think about arranging things and going out has actually improved my mental health (as weird as that might sound).
It doesn't bother me at all, I'm very self aware and I have ADHD, I know I'm terrible at maintaining relationships and low tolerance for BS so any whiff of drama or disrespect, I'm off. No headspace or attention span for it and I guess because I have that much going on around my brain at 100 mph ALL THE TIME I don't really have time to feel lonely a long with my family and everything else to take care of too.
I was hurt deeply by a best friend several years ago and I think a lot of how I am about having friends probably stems from that, but that situation also taught me that I wasn't always the problem either, but I am happy and content so I have no reason to change anything.
I know people think I'm a bit of a loner, on the school run, on social media all that stuff and I even make jokes about it but no one knows I have ADHD apart from close family. I know I'm a nice person and there's nothing "wrong" with me, I've just not met people who I click with.