Thank you for all the replies. I actually put off coming back to the thread today as I was tired and emotional and was expecting criticism.
To answer some of the questions, he has never slept well. Firstly due to colic, then sleep apnoea, night terrors, all of which he's grown out of. He'd a good talker and will say he's scared of being on his own/gets lonely, but occasionally will say 'I'm going to be a good boy for you!' and will sleep.
He dropped off at midnight last night, and has gone to sleep tonight at 9pm. I've always aimed for 7.30 as that seems to suit him best but that does vary.
I'm absolutely amazed that so many of you let your LOs in with you. Everyone I speak to in real life is horrified when I say I sometimes give in and sleep in his bed with him/on is floor or let him in my bed.
I think that is my problem, I'm constantly battling with what I think is best (I'd just co sleep if I had my way), and what everyone else tells me I should do. That makes me inconsistent in how I deal with him and I don't think the poor little bugger knows whether he's coming or going.
I think I will stop the battle and just let him in with me every time he wants to. If that doesn't work I will look at the medical issues as suggested.
Thanks so much to everyone.