Thank you for taking the time to respond. Although I feel more guilty and defensive after reading some of the comments, I am glad that you are being honest and it does help me to see others' opinions and realise that I am not being unreasonable.
Firstly, I've been asked how I could let it get to this situation and have it carry on for 3 years. I think there may be 2 reasons: I had an unusual upbringing myself. My mother has never been very maternal - no hugging or kissing, saying she loves me. She's still alive now and never makes any effort. I try to see her twice a year (her birthday and Xmas) but that's it. My parents had a violent relationship and they divorced when I was 9. I then lived with my grandmother and her sister, which I'd done every weekend before then too as my parents ran a hotel and had functions on each weekend. Also, I believe my DP has some personal issues which he/we need to address. Also, he doesn't want kids and so I wonder if he just tries to ignore mine as much as possible.
At first though, he did appear to make an effort and played with my DS, watched films with him, etc.. However, I now fear he only did this to make a good impression.
We have done things together - a holiday abroad, camping last year, walking the dog at the weekends - and I definitely need to encourage this. However, there is a part of me that thinks that deep down it's not what he wants and maybe it should come to an end. I do care about how it's affecting my DS, of course. He means more to me than anyone else and so my relationship with DP may have to end for that reason alone.
DS asked me the other day when DP was going to go back to his own house which is obviously very upsetting as it means he doesn't see him as part of our 'family' at all. DP only tends to get involved with DS is arguing with me and then DP will butt in to tell him to listen to me. So annoying!