Joining you on this thread.
I have a nearly 3 year old who has never consistently slept through the night. He appears to be largely immune to all know forms of sleep training. Much as you describe, OP and emmylou he wakes after midnight (bedtime is not always easy either) and kicks up a complete stink if not allowed to sleep with me (/and DH, but his ideal is to kick DH out and have mummy to himself). If we put him back in his own bed, he will contnue to wake through the night (citing spurious reasons such as needing water, the potty, to "just check" that his toy trains are OK, that one of his numerous soft toys which he sleeps with has fallen out of bed...) I am not all that consistent about not letting him come in to bed with me - sometimes I'm just too knackered and cave (but then still don't get much sleep, as he's a kicker and a wriggler).
I don't know where we went wrong either. We taught him (gently) to self settle from birth (i.e. we just always put him down awake from newborn stage, and he was able to get himself to sleep without fussing). We have a routine. I have a 5 year old who has slept through since he was around 6 months old (although he occasionally gets woken now by his marauding brother appearing in his room).
I cracked yesterday, I am so tired, and decided that enough was enough. Last night he woke at 3, wasn't allowed in our bed, and yelled the house down. Eventually I told him at that if he stayed in his own bed until the sun came up, I'd take him to the toy shop in the morning to choose a toy. I had to sit by his bed until 5am, but he did fall asleep on his own, and stayed there until 7.30. So we're off to the toy shop
. I am going to take a firm approach backed up by bribery (sticker chart, with maybe a small toy a week - I don't want to bankrupt us!) and just tough it out for a while. It really can't be worse than the effects of 3 years of sleep deprivation. We'd really like to TTC a DC3, but there is just no way I could cope just now with pregnancy or another baby, I'm so tired already.