I think it's a very personal decision @Howthehinchstolexmas
Lots of people end up in sexless marriages or with mismatched sex drives, and that can happen in any long term relationship, no matter how passionate initially.
One thing I would caution is not to make any long term decisions (marriage, children) until at least 3-4 years in, & probably after living with him for some significant time - you need to have all the honeymoon period gone, and know, without the friction-easing aspect of sex and fancying him, whether all those little irritating habits that every other human has, are going to drive you insane and make living together miserable.
But I don't think it's a deal breaker, if sex isn't that important to you (or him). For some people it is, others not so much.
Also, do you have any past relationships where you didn't initially find them attractive in all ways, but some aspect of attraction grew over time, or has it only ever declined over time for you? That might also be important to reflect on.
And think forward through life - mid-30s means you've got a lot of life left - would you be happy forming a relationship with him and presumably never having sex with anyone else ever again? How do you feel when you imagine that?
Lots to reflect on - also you're only 5 months in, so plenty of time to just see how it plays out for a decent while longer yet....