By that age, DS was having an hour nap max and bedtime was a ridiculous affair that had me in the state you were. He didn't nap much at all after he turned 2 (had to tell nursery to not even let him doze off as it had such a knock on effect on the quality of our evening).
Yes, he got tired later in the day but it was better for all of us.
Now, aged almost 5, he goes up at 7, bath/ shower/ arse about, read together then by himself, lights off at 8. Usually asleep within 20 min or so.
Then mostly nothing til I have to pretty much shake him to wake him from the sleep of the dead at 7:15 for school (either because I am then leaving for work and I like to see him first) or because he is such a faffer that we need the hour to actually get him to eat, get dressed (yes that order to avoid yoghurt down uniform etc) and then, by his preference, have a bit of a play.
Sometimes he has a bad dream, or wets through the nappy pants (no sign of them being done with any time soon but honestly, fine with that) and then comes into the spare room with me, but he just turns over, goes to sleep and not a peep from him, usually. I don't very often get punched by accident. DH is also acceptable at night these days (and sometimes gets called for rather than me, which is bliss)