The stress of taking him to the vet now would be too much I think, unless you absolutely have to for an illness.
I second Feliway. I'm not 100% convinced it works but could be worth a try.
Will he come near you for some extra special treats, say if you sit on the floor? Maybe he doesn't realise you actual want him near you? Sorry, I'm just thinking out loud ...
Can you tempt him out to play? With a Da Bird or similar? So he sees you're a playmate?
I had a rescue cat when I first moved out to live on my own. He stayed under the bed for ages, about two weeks, but used to come out at night to explore and eat etc. Then eventually he'd be on the bed asleep when I woke up.
I've had my rescue Siamese from the age of 13 weeks. But he was very feisty, played frim the word go and was on my knee asleep the second evening.
I know it's frustrating, but he'll get there I'm sure.