vimto, you said 'because his relationship (with his ex) was non-existent, he focussed all his attention on his kids'
What if you have been given the wrong impression here, what if his relationship failed and was non-existent BECAUSE of his focussing all his attention on his kids?
He certainly seems to be struggling as a father/partner in general here, and reverting to what he knows?
I'm sorry to say that the old adage, 'People don't change' holds true, you went into this knowing it was flawed and hoping he would change, men like this very, VERY rarely do, and he has serious amounts of previous form. It's OK, not judging you, we have all done it, our eternal hope in characters like this is what has kept the human race going 
You need to take a firm approach to this, if you allow him to carry on, it'll only get worse and ultimately fail anyway. If you do nothing you will pretty much condemn yourself to a lonely and single-parent style life within a relationship, being over-ruled by 3 stroppy teens (including your DP here). DP's never going to suddenly wake up one day and resolve to be a great P/Dad to your DS, unless he is kind of given no other option.
If you take a strong approach now, it can either remedy the situation or bring it to a natural head. At least if you stand up for yourself, whatever happens, you'll be fighting for your DS, your own self esteem and place in life.