"There may be some men who aren't fussed about sex but they are very few and far between."
How can you talk for the entirety of men? Because YOU are fussed about sex, ergo ALL - or "most" - men are just like you? This is a logical fallacy called the bandwagon fallacy.
You have NO idea how many men are asexual/low libido. No one does. WHY?
Because even admitting to themselves that they're not that keen on sex is forbidden to men, because otherwise they're "less of a man", a "half-man", a "non-person", and they feel ashamed.
There's a lot of posturing and jostling between men, competition and aggression, it's not safe for a man emotionally OR sometimes even physically to say something as iconoclastic as, "I don't want sex" or "I can take or leave sex".
So NO man is going to tell another man any such thing. They're not going to dispute the utter importance with which you hold sex either, for fear that they'll be deemed a pansy or loser. They might even join in with ribald jokes etc. In fact, they are probably more likely to lead with such sexual language to disguise their shame that actually, they can take or leave sex or OMG they don't even really like it.
This groupthink monolithic talk about men by men does a disservice to men.