OP, lots of people are referring to the boy as your stepchild (which you, wisely, did not).
It's perfectly normal not to love someone else's child. There are many, many times since I had children that the only thing that has kept me putting one foot in front of the other is that I love them viscerally because I gave birth to them. I wouldn't feel that way about someone else's child, however fond I was of them.
That being the case, I absolutely would not countenance dating a man with young children. I don't want to do the 'hands on' parenting of someone else's child, or be constrained by them, or have to go to theme parks/insert whichever child-friendly activity for the sake of someone else's child. I don't want people groaning at whatever I've made for dinner, and needing to be helped to go to sleep, and needing to be told to come off their devices.
The difference, obviously, is that I've done all that with my own, and there is nothing that would possess me to do it all again. I would automatically not start anything with a man with young children, because any decent man will put his young children's needs first in a new relationship - which would mean I would always take second place. And if he wouldn't do that, he wouldn't be a decent man, so I wouldn't want a relationship with him anyway.
OP, this six year old comes as a package with this man. You have to decide what your role is going to be. There isn't any reason why you should have to spend 'his' time with his son; the two of them could perfectly well spend it on together. You're not the boy's stepmother, and there's no compulsion for you to do any of this, so long as you're friendly and polite when you see him. Or you can try to become more involved in a positive way - though I can't see that working as things stand. Or you can end the relationship and find someone who doesn't have young children, taking this experience with you as something you don't want to repeat.