I was in a similar situation a few years ago. I'd come out of a relationship and was lonely, low etc and began talking to a guy. He claimed he wasn't just interested in sex but that was literally all he would talk about. Whenever I tried to talk about anything...real (for want of a better word), he would just try to change the subject back to sex (whilst still trying to deny that he was just interested in sex). I may not have been in the best place but thank God I was sensible enough to realise that, despite his reassurances, he was only interested in sex, and as I don't do the casual thing, it would have definitely been setting myself up for a fall if I'd met up with him. He sounds similar to your guy (so to speak) - never responded to texts unless it suited him etc. The only difference, from the sounds of it, is that I knew he wasn't the type of guy that suited me...but darn it, if he didn't have a pretty face.
The last conversation I had with him, about 3 years ago, he even said that if I was with someone, he'd still want to be friends (because, of course, he wasn't just interested in sex)....then when I said something along the lines of "That's good, because I've actually just started a relationship with someone...." he swiftly disappeared and haven't heard from him since!
I think, if it wasn't for the fact that my circumstances changed over the time I'd been talking to him and I was in a much better place, I think I would have felt similar to you (and have a number of years ago, with a different guy that I thought something was going to happen with).
If he's not genuinely interested in you though, it is much better to shut down contact now than keep it going and get even more invested.
Looking back, even aside from the one track mind, I had a lucky escape...he already got controlling when I went out and got back later than I said so we didn't end up talking until later and there were other weird instances that it was only later that I realised were off that and the fact his dad was a complete creep
Keep your head up, it will get easier with time!