YANBU. Even if that's genuinely been your experience, its not helpful to say it to a Mum of babies or young children who is struggling.
I have two DC (9 and 5) with SEND. Although their needs mean they both probably require more 'hands on' parenting than other children their age, and some days are still very challenging, life has definitely gotten easier and more fun.
The daily challenges of parenting are much easier to cope with when you're not chronically sleep deprived for a start. DC2 woke every 1-2 hours and was up for the day by 5am for the first 4 years of his life. Now, he has the occasional bad night, but he mostly sleeps through which has improved all our lives immeasurably.
The DC's SEND means that we still have to be very routine-driven, events have to be carefully planned and we have to anticipate their needs in various situations. But we are used to doing that, because we've always done it, and not having to faff about with changing bags, pushchairs, bottles and nappies definitely helps when it comes to things like trips out. When they were toddlers I dreaded going out with them because of the potential for explosive tantrums and the herculean effort it took to get them both ready and out the door didn't seem worth it. Now we understand their needs better, we can predict when things are likely to trigger them and have strategies to deal with it, so the meltdowns are much less frequent and days out are mostly a pleasure.
Of course, there will be children with SEND who have much higher support needs than mine, and I am also fortunate to have a supportive DH who does more than his fair share, but I do think that life gets easier for many parents as DC get older.
I work with teenagers so am under no illusions that those years won't be challenging. But nothing could persuade me to go back to the relentless physical exhaustion of the early years where I couldn't sleep for more than 2 consecutive hours, finish a meal or shower more than once every 3 days because they were totally dependent on me to meet their most basic needs.
In the last year I have started to read books again, I have the energy to socialise with friends in the week for the first time in years, and the last family holiday we had, (although I wouldn't go as far as to say it was relaxing) was definitely fun rather than feeling like a test of endurance! I feel like I am gradually getting back to myself.