OP, I'm wondering whether the problem is that you are too involved and trying to hard. Doing so can result is the same frustrations that if you were unpleasantly ignoring him.
It's a very very hard situation and I find it very admirable that you stuck by your man despite it all and so desperately want to be a good SM to your SS who you could, without helping it, resent dreadfully.
The issue though is that from the perspective of the mother, and probably your DSS, and even maybe your OH, this child comes to your house to see his dad, not you. From the ex's perspective, it must have been hard to allow your OH to be part of his son's life, I expect it would have been much much easier for her to pretend he wasn't there and raise her son without his involvement, but she -hopefully- realised how important it is for a child to have both his parents in his life, however, that doesn't extend to having a SM in his life. Similarly for the child, it might very well be that he is only happy to come over to spend time with his dad, not you.
He has now been seeing him regularly for 3 years, so really it should be getting better in the child should start considering you as part of his life too, however, if you are too expectant that way, it might be having the exact opposite effect?
If that is the case, then the best thing you can do is to take a step back. Don't reject him, don't show resentment, but do allow him to have a relationship with his dad without your involved in every, or almost every step of it.
Hopefully time should help as he chooses himself to grow closer to you.