It's a difficult one. I'm the light sleeper, stressed at work one and as I need to get up at 5:30, need plenty of sleep, and to read before going to sleep, I got up to bed around 9:00, un bed by 9:15-9:30. The problem is that I normally read until my eyes close and I naturally fall asleep which is between 9:45 and 10pm. The problem is that OH normally comes to bed at 10pm, that is just when I have just drifted off and therefore easily woken up by any noise/movement and then once I'm awake, it's almost impossible to get back to sleep right away, so I'll see my OH gently going to sleep whilst I'm wide awake for another hour or two. I do use ear plugs, but in my case, it's more the movement on the bed as he gets in that wakes me up than the sound of it.
If I can be asleep for 1/2 hour by the time he comes, then I'm usually ok, so in my case, I would like him to come at 10:30, but he gets tired and wanting to come to bed at 10pm, so not fair on him.
I usually go along with it until I am so shattered, I started getting stressed at the prospect of another bad night and I end up in the other bedroom. Going straight to sleep makes a big difference as I don't anticipate being woken up and that makes falling asleep much more relaxing. At the same time, I can't envisage not make it permanent as I do like the feeling of sleeping next to my OH.
All this to say, I understand how your OH feels, but it doesn't mean that he is right to demand you go to sleep when it suits him. It sounds like you have agreed to what suits him, and therefore is not fair that he should give you a hard time the occasional times when you don't stick to it.