The initial stages, baby, toddler, young child are the most stressful, although when they hit the young child stage, you do feel like the pressure eases somewhat.
I think the first 3 years are definitely the hardest, possibly longer depending when you potty train.Theres others who say when they go to school, that's a definite benchmark, where it seems to get easier as they get more independent.
Each age level has its fun and games, a pre pubescent teenager and a teenager make for a lot of fun, with attitudes and atmospheres, the children wanting to be pretty much left off the rope. Our eldest has either an amazing memory or has kept notes, if we let the youngest do something earlier than we let him, the cheeky sod asks for compensation!
It depends also if you love the baby days, I know some people adore the baby days, they seem to lose interest as the child gets over. For example baby days, toddler days, you'd have none stop photos and videos, from being on the breast, to baby wearing, to baby led weaning. You notice it more in families who have more children, you rarely see the older ones.
There's those who love the later stages, lots of clubs, lots of doing different activities. Lots of achievements. Although there's achievements in the baby stages, there's so many different achievements in the older child stages .