Sorry that this happened. I can see why you're anxious.
But there are so many variables here. First, even if you do have it, will you know?
Recently large numbers of my students have tested positive. (We have an in-house university testing service, so they don't have to have symptoms.) I've found it quite remarkable how many have no symptoms at all, and how many of those who do have symptoms test negative (and have bad colds/ freshers flu). You might think this this is just because they're young. But two colleagues in their 40s/50s have also had asymptomatic positive tests.
Second there's the question of immunity. It's not impossible that you've already had it. Or even, though very unlikely, that you gave it to your colleague! I realize that there's doubt about the length and universality of immunity. But have to say it's been working in my case. I had it in March (confirmed by antibody test) and haven't managed to get it again despite being several times in similar situations to you, or even closer/ for longer. Or maybe I have had it again but just don't have symptoms? I haven't tested again.
Third, the possibility of false negative test results if taken on the wrong day, not swabbed deeply enough etc.
So not only is it unclear how likely someone in your situation is to catch it, there's also a fair degree of uncertainty about whether anyone will be able to know you have if you have (and haven't if you haven't). All of this is in favour of it not being a terrible catastrophic disease for most people, of course, so although the uncertainty isn't exactly comforting in itself, I think it should reassure you that even if you do get it, you have a good chance of not being very ill at all.