ali23, mine also turned into a dream sleeper around 17 months. After I weaned her from bf-ing during a nursing strike - so no tears from her.
Prior to that, co-sleeping was my survival strategy.
Having had two poor sleepers (and still in the throes of one who is teething 6 teeth at 21 months), my theory is that for some babies:
(1) Their minds race and are difficult to change gears, hence troubles with settling and fighting bedtime despite being overtired and hysterical.
(2) They are light sleepers, hence the frequent wakings.
(3) Their first instinct when they are disturbed is to spring themselves awake (rather than just go back down). My dcs are very cuddly and I put this down to a form of separation anxiety.
(1) is a function of temperament. My dh is still like this and cannot just put head on pillow even if very tired. He needs to wind down first. Although they can learn winding down strategies over time. My dd 4.9 still needs a lot of help to wind down. She falls asleep on our lap on the sofa after dinner and we carry her up.
(2) - the good news is that they will grow out of this to a certain extent. For reasons of survival, babies have short and light sleep cycles. This is because in the open their survival absolutely depends on not being separated from their mothers and waking for night feeds. As they mature, could take a year or more for crap sleepers, their sleep cycles lengthen with longer periods of deep sleep (bliss) and it takes them a shorter time to go from light to deep sleep.
(3) At some point, the baby will realise that it is not in his interest to wake himself up. And to go down again. For some babies (mine and yours), it does not happen naturally. Once they realise this, I think this is the magic switch that makes them finally sleep through, assuming (2) has been achieved. With my dd, it took weaning from bf-ing.
Not sure with ds though. I remember the turning point was when I felt that suddenly, dd's sleep was more robust ie she did not wake when I carried her from one place to another. She did not wake if I touched her.
And also don't forget, it is always darkest before the light ...