But the OP has already said she wouldn't have been particularly interested herself.
Look, I am arguing for for DP because this has gone one way and I feel it could be genuinely innocent. My DP grows and sells rare plants and will quite often give away cuttings or blooms to 'other women' (or men but there are a lot of ladies into the hobby) at the slightest passing encouragement. He will post them to Facebook strangers even as well as folk at work.
We have a house full so I'm not bothered. A fossil does not take any looking after. Who's going to nick it? What's all this about he's taken care of it for a week, what in his underpants?
I can picture a man like my DP, enthusiastic about his plants/ fossils who has taken a colleague's brief interest with innocent pleasure.
I could be wrong, and suggested something to say earlier but OP to make clear she isn't happy, but she hasn't mentioned any other suspicious behaviours.
The woman at work doesn't have to have a known interest in fossils to have passed a quick comment. It's a 2 minutes chat. 'youre off on a field trip then? 'yep, to X looking for Y' ' how nice! I had a couple of those as a child!'.
OP what is the bigger picture?