Oh OP. Whether this oh-so-slowly-man is the right one for you, you MUST, absolutely must make a life for yourself. Have your own interests and things that you like to do, separate and away from him.
Make some new friends, not bosom buddies, no need for that - just people who you like and who like you enough, to go for coffee, go to the cinema, bowling, swimming, whatever.
If you keep being available to this man, he will KEEP you at arms length. You will keep tying yourself in knots about him, being totally in love with him and he will abuse that because when a person has the one-sided devotion of another, they cannot help themselves but test it. You will be unhappy with that testing.
You're pathetically grateful for the crumbs he throws you because you haven't anything else in your life. You're worthy of so much more. If you carry on the way you are - being so obliging and grateful for every little consideration - you will lose him. Personally, I think that would be no bad thing but it will hurt you and I don't want that. You sound nice. You deserve better.
Be unavailable - genuinely - go and do something for yourself and let him wonder where you are. Don't be in a rush to text or return phone calls. I guarantee you would be a lot more interesting to him than you currently are.
The leaving you out, bearing in mind he knows you have little family/friends, is quite a spiteful thing to do. It was done on purpose and whatever role you think you have in his life, he doesn't think the same and has probably told his friends that it's not serious.
We really ought to have a thread about being a bit more circumspect and unavailable to people who take us for granted - not just boyfriends - anybody who does this. Swapping stories and tips to build our own lives up would be comforting. I wish I'd had this years ago when I prostrated myself at the feet of the most awful man. Urgh.
My granny would have told you (and me, if she'd been alive then) - "Love them enough but not too much". I think she was right.
Finally, I saw this on another thread, from another poster. I don't know what it means exactly but I think it must be empathetic and soothing so... "Howay man, Lass". :)