When he says he likes to be with you, and thinks about you all the time, or says that you see each other "all the time" (meaning, between 5 pm when he gets home from work, and 6 pm when he goes out to dinner with his friends), I think what that translates to is: he knows you'll be available to him when he wants to see you, so he feels comfortable with his access to you and doesn't see a need to maintain the relationship with active effort. In other words, he is taking you for granted. That is how I interpret his behavior.
I don't want to judge him, I don't know this man. But I think it is true what someone said upthread, that you know someone by their actions, not their words. I don't really count the scattered hours together when a person is returning from work and heading to another activity as "time together" because it's not quality time that you are actively paying attention to each other. I see the same grocery checkout clerk at least a couple times a week for at least 5 minutes. That doesn't mean we are friends. It doesn't mean I feel like our relationship is valuable to her. We just have interacting goals in the same place on a regular basis. He chooses to spend his free time with other people, not with you. So what he really enjoys is being with other people, not with you.
I don't really buy into the whole idea that him spending time with his friends over you is "selfish." He's doing what he enjoys. The problem is that he enjoys is being with them more than being with you. You enjoy being with him rather than being with your friends. The relationship is asymmetric in that you want to spend time with him more than he wants to spend time with you. He could spend more time with you, if you force the issue, but would he feel like it was a sacrifice, that he was giving up what he REALLY wanted out of guilt? And would you really be happy being with someone who felt that, by doing so, he was giving up what he really wanted?
I'm actually married to a man who is a bit like this, although it is his family rather than friends that he enjoys being with. We've worked it out now (or, we moved away from family for work and he has no choice but to spend time with me because he doesn't have family or friends here) and I would say it's actually a bit soul-destroying to be tied to someone who sees it as a sacrifice (rather than a joy) to spend time with you. It just chips away at your self-esteem bit by bit, to feel needy or nagging for wanting the person to actually WANT to spend time with you. (Except when sex is desired, then you're interesting. After that, you're back to being boring again.) So, if this were me, I would run.
But, it's your life. Only you know this person and what is really going on in your relationship.