Everyone should be able to go out for meals....and fortunately there are a range of places available so that everyone can. However, they aren't all really suitable for everyone all the time. Whilst anyone is free to go anywhere anytime in theory, in reality, considering what you and your party will enjoy and also who else is likely to be there and if your presence will impact their enjoyment just seems sensible.
Many more places are suitable at lunchtime than evenings for small children. They are more relaxed, less stuffy and lots expect families to be in. Some still really aren't suitable often because they are quite stuffy and very quiet places and small children aren't very quiet usually. Most families wouldn't want to go and could easily think of lots of places with an environment more suited to their children, meaning they leave those who like the very quiet restaurant to enjoy it in peace.
For those with noisy children or those with needs which mean they might not be so quiet, who struggle with the idea that they are 'not allowed' to go to nice places or people are trying to prevent them...does it help instead to think of it more from the angle of whether those environments are such that YOU and your children will enjoy yourselves? Don't we all enjoy ourselves more when we feel comfortable, and part of that is also knowing the environment suits us and all the other diners there? Personally, my enjoyment is reduced when I feel other people are behaving extremely differently to me and particularly if I feel my groups behaviour is impacting them negatively. As an example, I avoid large, loud, busy pubs full of drunk people, because I don't enjoy them as I'm not usually loud and drunk. I also avoid pubs with sport on TV as I will want to chat to my friends and e chatting can be annoying to those watching the TV. I go to intimate, quiet restaurants when with DH alone or in groups of adults, finding there are other groups of adults and everyone is enjoying some child free time. I go to chain restaurants, often at lunchtime with the smaller children, knowing there will be ither families and people and staff are relaxed and a bit of noisy toddler or constant going past to the loo doesn't phase anyone. I take the older children to nice restaurants at lunchtime....they are able to sit quietly and enjoy the good food for a couple of hours with no trouble and we all enjoy it and don't impact others. So I consider the other people likely to be there and also what we will enjoy.
With very small children, I think there just is a phase where some of the nicer restaurants at nighttime have to be forsaken for a short period. Honestly, it's not asking the world to not go to them for a couple of years or to only go when on a rare child-free session. I don't like the entitled attitude of some parents with small children, that they will do absolutley whatever they like, regardless of the stage and behaviour their children are at and regardless of how it might impact anyone else at all. Children bring some limits to life....most things are short-lived phases, but just part of it.
I can see that having children with additional needs, who will always have them as they get older is difficult and potentially more restrictive long term, and that long term potential is what makes it especially hard. I guess that in reality parents in this situation consider what they will enjoy themselves and to be honest it's hard to enjoy an expensive meal in an intimate restaurant where other diners are very very quiet, if your child isn't quiet at all. You may be totally entitled to be there and to enjoy a luxury meal just like anyone else and your child might be too, but would you actually enjoy it, beyond the issue of making a point that you are entitled to be there?
I've been to restaurants /cafes where there are groups of adults with additional needs who have been brought out by their carer. They are usually lower-end, relaxed restaurants. Often the adults who have been brought are being fed by their carer, there can be lots of noise and mess....but everyone is having a good time. Of course they should be able to enjoy a meal out like everyone else....but the careers don't take them to a small, quiet and intimate restaurant where people spend £100+ per head on a Friday night. They could but they choose not to for the enjoyment if all concerned.