Very worried.

[Mummyto2girliesxx]

Been with dh for 15 years we have 2 beautiful girls 10yo and 4mo

We've had our ups and downs but everything is now very fantastic .... we've had our long awaited second child thanks to fertility treatment and have felt happiest I've been in a long time. (Still have rows but who doesn't?)
Anyway about 6-7 years ago an ex boyfriend (not serious) contacted me on fb and at the time me and dh were not in a good place, rowing all the time , not good financially, I wasn't happy at all. I don't think dh was either.
Anyway, these messages with ex went on for about a week, mostly just general chit chat and him being a bit flirty telling me I looked good in my pics etc (massive confidence boost for me) I can't actually remember the whole conversation as it was so long ago and it meant nothing to me. Anyway fast forward to a few weeks ago I went to a cousins children's birthday party and this ex was there with his children , turns out he's friends with my cousins dh all fine, I said hello and that was that. however now I'm worried that he's told cousins dh about our fb exchange all those years ago. And let me say now , cousins dh is a massive gossiper and would love to have this information about me.
When I visited their house last week to exchange xmas presents we was taking about face book in general (I'm no longer on it , haven't been for 4 years) and then cousins dh said , facebook is evil and a family wrecker while staring directly at me. Now I don't know weather I'm being paroinoid but I thought he was implying that he knew about mine and ex fb messages.
I felt quite freatened. Now I'm so worried that cousins dh will somehow make sure my dh finds out and I know for a fact our whole marriage will be over. My dh hates social media and we have had countless rows about it. I know for a fact that that would be it. Now I'm worried.
Am I looking too much into it ???
What should I do??