Im married but unhappy and not in love. 
I met a man in a shop where i work and we swapped phone numbers. he came to visit me and within mins asked me for sex. I got freaked and said no.
You gave some random bloke your number and was shocked when he thought you were only up for a shag? What were you expecting? Did you think he was going to court you?, or even perhaps propose? 
He came to visit again after i called him to say sorry. this time i came on to him but he freaked out!!
Not content with humiliating yourself ONCE you went back for more cringe. 
now he avoids me and wont come into the shop ..erm [no shit sherlock emoticon]
Help whats happened you need to ASK??? seriously? 
I really like this guy. WTF You are supposedly MARRIED!
You may not be young, but you sound it.
You may not consider yourself dumb... But come on, ask yourself, what opinion would YOU form of a woman who gives her number to a random and then is surprised that he only wants sex.
You are unhappy in your relationship? DO SOMETHING ABOUT IT! Don't cheat FFS, only really shitty people cheat.
He cheated on you in the past, LEAVE HIM.
It really IS that simple.
If you have bargained away your dignity for the sake of a house, a ring on your finger and the title MRS as opposed to Ms, then that is YOUR decision.
Those that cheat on their partners have no morals, so don't be surprised if you chucking your number out means that you get treated as someone who'd only be good for a shag.
Find your dignity, find your happiness.