It definitely feels like that but I think it could be down to other factors.
When I left (a rather posh) school at 18, I bucked the trend and, rather than going to university, buggered off to Butlin's and became a security guard. Basically this involved standing on doors and patrolling the site for wrong 'uns. The pay was rubbish, the hours were long and the weather was, by and large, cold and soggy. However, I remember it SO fondly.
I now get paid well, have a nice large office to myself with all the 'mod cons', decent hours, pension etc. but still hark back to my first 'proper' job and how much I loved it.
Was it because I was younger, had less responsibility, was meeting more new people and getting laid meeting girls (a relative phenomenon having been to an all boys school).
I think sometimes we look back on old jobs with rose tinted glasses because we are actually looking back on our younger, stress-free selves rather than specifically at the job itself.