Stop calling yourself ridiculous. Stop thinking that you're over cautious. Stop wondering if what you're feeling/doing is normal.
You're not here to 'follow the rules' or 'get it right' or 'be normal'. Ideally you want to find somebody with the same sort of weirdnesses as you, so the parts of you that are 'normal' or 'average' or 'standard' aren't really the relevant ones. We all have them, and we all have strange aspects. You have to find someone who matches you, not act like a 'normal' person so that you can meet another 'normal' person. How tedious would that be? Especially if you're looking for a partner for life.
There are no rules. Your feelings are signposts, to be respected. They're not obstacles to be overcome in order that you can have a successful relationship/dating experience. Think about it: you're not looking for someone to tick all the boxes, on paper. You're looking for someone who makes your feelings soar, so your feelings have to be your guide. Nothing will soar if you're thinking 'I don't like what he's doing, but maybe that's because I'm abnormal..?'
Feelings soar when they get better and better. So anybody who gives you a negative feeling... just move on. It doesn't matter why. There's no logic to feelings. They're your wild bit. You just have to observe and respect them. And deal with them in all their wildness, as an adult. Get used to doing this. This hearing and responding in an adult way to your wild feelings. Once you've got used to that, start dating. You'll find it much easier to say 'Nah!' and walk away, if you've got used to listening to your heart and your gut. You'll be less anxious; anxiety is a feeling of not being rooted, of feeling like things are right on the brink of flying out of control... but if your heart is guiding you, and you are actually listening to it and respecting it, you'll know that you will never have to endure an uncomfortable situation, because if you find yourself in one, you'll simply say 'I'm leaving', and that will be it.
And that's the thing with dating. Meet up with people for 'a drink'. Resolve, to yourself, to stay for half an hour only, unless you really really want to stay for longer. A date doesn't have to be a big, whole evening, committed business. Go for a drink. Act like you've met in an airport at the bar and you're just having a casual chat before going your separate ways. Walk away. Over and over. One day you'll come across someone you really don't want to say goodbye to.
There's no 'right' or 'normal' way to do this. There's no right or normal way to feel. Allow yourself to be you. Allow yourself your feelings, all of them. They are your leader, and they are all your boundaries. They are your power, and they will take you where you are safe and loved. That's why you mustn't question or criticise them. It's the deepest form of self disrespect.
It's good that you said no because you felt anxious. And it's interesting he turned out to be a little weird... See? Feelings know stuff. I think your intuition is as sharp as a pin, and it gave you a message, and you listened. Feels good, doesn't it?