I totally feel your pain OP.
Dh does this as well, although he has thousands of his own hobbies.
I was always the long distance runner - ploddy but dedicated, but as I got older decided I didn't fancy the longer runs and discovered parkrun which is a weekly timed 5k.
Parkrun is amazing - I love the camaraderie, the cheers of encouragement even if you're at the back, and I loved the headspace of going there alone. For me it was never about getting PBs - great if I did, but actually committing and going there was the thing.
Fine until DH decided he wanted to give it a go as well. Fair enough but then he'd moan about how slow he was ( good 5-10 mins faster than me) and one time he had a dramatic gasping fit at the finishing line as he had pushed himself too hard. So the focus was always on him and how hard he'd pushed himself and what he wanted his next time to be.
Did it a few times with DS but he got bored, so then we had to take turns to go to it as one of us needed to be at home with DS. As soon as I'd get in from my turn, he'd be asking what time I'd got and I'd feel down if it wasn't a fast (for me) one.
I had to totally change my focus as I was getting distressed about my slow times and how rubbish I clearly was compared to everyone else. So now instead I have a goal of running 50 by the time I'm 50.
In the meantime DH lost interest as it was a bit too repetitive for him and not enough opportunity to buy new things and has moved on to climbing mountains again, which I'm happy to leave to him.