So he basically spends no time at all looking after the kids, properly, even the one day a week you work and he doesn't? He basically never sees them in the week as he comes home so late, and still goes to play sport on a weekend? And he thinks you going to bed early is the problem? I'd be tempted to say if he doesn't see it as a problem he spends no time with your kids, why should you see it as a problem you don't spend time with him? He seems to want you to be a full time nanny, cook, maid, cleaner, household manager but still a fun wife too, but expects to bring nothing to the relationship himself except money, not even care or basic respect, judging by your updates.
Also, when you say he has NEVER! got up with the kids because he doesn't wake up, i'd point out if I had the luxury of never waking up in the night, I'd probably stay up later too.
8pm bed does seem early but life is tiring with young kids, night waking and a job. And let's face it, your updates suggest he isn't much fun to be around so why would you want to wait up for him. The only reason I stayed up any later at that stage of kids and work was because I valued the child free downtime to watch TV etc, not because I wasn't tired! And that was with a dh who pulled his weight and was also part time. If your relationship sounded worth working on I'd say make an effort to stay up a couple of nights a week, but frankly from the way he speaks to you and points in your face (and then somehow makes that your fault!), why bother making an effort for him!
And please don't feel bad about the finger thing. Yes 'hitting' it out if the way makes it sound bad, but he aggressively hot in your suave first so you have the right to defend yourself and if I'm picturing it correctly, what you did would never injure or probably even hurt someone.