I think it would depend on the reason why they didn't drink, more than anything.
If they didn't drink because they had issues with alcoholism, or have done something truly dangerous or awful under the influence of alcohol leading them to stop, that would bother me. I wouldn't rule out a relationship though. However, I have been in a long term relationship with an alcoholic/serious on/off 'problem drinker', and I will never forget that even during the good times when he didn't drink at all for years, there in the back of my mind I always knew that the drinking cycle would begin again... just one drink to celebrate something amazing, followed a few weeks later by him having just one or two drinks when out - of course stating this wasn't going to be a new habit it was a one off thing, and so on until within weeks he would be drinking himself stupid every night, not coming home after work all night, losing jobs, arguments, insults and violence, money and savings gone, all over again, til the 'teetotal' regretful stage started up again... I would worry that this would happen again tbh.
If they didn't drink for religious reasons, fair enough. But I am an atheist and couldn't really imagine getting very involved with someone who was very religious. But you never know unless you try it?
If they just didn't like alcohol, or the feeling of being drunk, or they couldn't drink due to a health condition or medication they took, not an issue for me at all.
I wouldn't worry about getting tipsy when they were sober, as I don't drink very much, or very often. I probably have a glass with dinner once a week or so, and if I go out with friends to the pub it is only around once a month this happens. I always stick to two or three drinks, and I know if I have any more than that I will feel dreadful the next day - it really interferes with my sleep and I get wicked hangovers from not very much drink!
I certainly wouldn't tell someone they needed to start drinking, pester them and then dump them for not drinking! that screams 'so boring can't think or anything social to do without alcohol's involvement' or 'drink problem and bully'!