Wow. No, you are not being unreasonable.
If this is the way things are, then I'll be blunt: you need to split for the sake of both your children.
He seems to want to piggy back on the 'family' in order to be dad to his son - but isn't interested in you, or more importantly, accepting your son and his child's brother as a member of his family too.
Your older son will be utterly damaged by this. If there's going to be a male parent figure in the house, then that person needs to be HIS parent too. Not some bloke who makes it clear the division between him and his brother. And it's just as bad for your other son who also doesn't need to have someone driving a wedge into the family, treating him differently to his sibling, not being a proper 'dad' in the household.
It also goes without saying that a man like this isn't worth having anyway! What kind of good dad treats two siblings differently, just because one is his biologically and the other isn't? What kind of good partner acts the way you describe?
Get rid. He can be a father to your younger son from afar, which will be better for all of you. You're clearly a family unit which this man doesn't want to be part of. Fine, but he doesn't get to be like that from within your home and cause sadness, bad feeling and division.