I am a 1st September birth too, and while I think I started school at 5, a few of us were moved from Reception to Kindergarten early, so I ended up in the year above. I wasn't the youngest though. One poor girl was born in January. I used to think she was a terrible crybaby, but looking back, she really was virtually a baby.
None of us did badly academically - I guess we were picked to go up because we were doing OK, but I think socially I was definitely behind the older ones, as I would have been if I was August born, of course, but then there would have been no choice. My mum told me not that long ago that she could see a difference in maturity between me and my peers even at university.
It was a bit of a drag being the last one to be able to start learning to drive at school, but other than that I don't think I found it that difficult being younger. Now it's great, as I have nearly a year on most of the school friends that I still see. They were contemplating their 50th birthdays when I was still a 48 year old spring chicken!
However, watching my May-born son struggle behaviourally an socially, where my November-born daughter flew through primary, I do think a later start gives you an advantage overall.