I always wanted 4. (recurrent miscarriages and perimenopause put a stop to that ever happening.) However I am a foster carer now and so we often have 4 or 5 children living in the house, and it can be busy.
You do have to plan one-on-one time with your children more, when there are more children in the house. But they value that time more too.
0-1 and 1-2 were definitely harder than 2-3 or 3-4 or 4 -5.... I think once you have gone past two children, you know that you only have one pair of hands and one pair of eyes, so keeping a watch on three children is tricky and more is harder still, but the older ones start to watch out for the younger ones, and the characters of the children make a huge difference on how easy or difficult it is with more children.
I would say that having a child with special needs is more of an impact on the family than the number of children. The more the merrier in this house!
With the ages of the children that you have, just be prepared for the terrible twos of the youngest to hit, at the same time as the terrible teens of the oldest.......... The combo is a killer.
Depending on your own age also consider what if the peri-menopause hits at the same time as teenage years for your youngest.... but hopefully you are wonderfully young and that won't happen for you.
Having 4 children is easier when they are younger in my opinion, but perhaps that is because we are in the teen zone here (with baby foster children at times).