I've been in your position.
All @Nattyjackie 's points were true, and also:-
Where as I have to loose the thing that was keeping me sane in lockdown.
Think about it a different way. When you look at it you can realize this 'relationship' was actually causing you pain each day. You truly were lonely, away from the person you liked. You were not truly with him for a lot of the time when you wanted to be. You feel sad now and a bit bereft, but the loneliness of being sat alone until the bloke can be arsed to contact you is actually worse.
I've lost someone who loved and cared for me
He didn't really Not in a significant way anyway. Yes, he spent some time on/with you but not at other times, and at the end of the day he chose someone else, leaving you hurt.
someone to share my day with and spend peaceful time with.
You never had that with him, at least not most of the time.
He still gets that even if it's now built on a lie
What he had with you is what was built on a lie unfortunately, clearly as can be seen by his actions. He didn't love you enough to put you first, whatever his excuse.
What helped me after I dumped him not to love the married bloke at all, was to realize I can never know what was the real him, I can never trust him as I don't know what was fake. This meant I had no love for him anymore as it could all have been fake for all I know. And I started to realize some of his motives/character.
I know that's not the same situation, but you could take some of your power back. Block him on everything if he hasn't done that already- FB etc, phone, email, everything.
You may know of previous affairs he had (or maybe he told you you were the only one.)
I would consider telling his wife somehow, if you feel she has a right to know.
And I can't do all the things I would instinctively to to get over it,
Lockdown has eased a little in most places, so you can think of some things you could do; visit friends in their garden and have a drink, BBQ or whatever, have a wander out social distancing with a friend etc. Pick up takeaway coffees and sit in the park with friends. Chat to friends on messenger/video chat. Anything else you can think of.
You may find it helpful to join FB groups for OW's (they will be closed groups so no-one can see you've joined) or forums. This forum is quite low traffic but I found the people there helpful. community.affairhealing.com/
My 'ex' was actually a very nasty narcissist which I didn't realize until the end, with the help of MN.
It might help you to watch YouTubes or read stuff about married men or narcissists just in case he is one.
Look back and think of all the many times he hurt you/you were in pain during the relationship.
Best wishes and don't do it again. 