I would definitely tell him.
Your anxiety affects your behaviour. It is likely that if you don't tell him, he will be confused as to why you are acting in a particular way and might start thinking that it could be something that he has done/said that's affecting you or that you don't feel comfortable with him.
Better to be open now and explain calmly what you feel and how it affects your daily life.
Some men won't be able to cope, others will be understanding. Better to know now than invest in the relationship only to have him disappear in a few months.
You have done nothing wrong and you did not decide to be affected by anxiety.
If you avoid him without explanation, he will simply think that you don't want to see him and you are likely to lose him.
Ignore the people who are telling you it is too soon.
I have issues with mental health/childhood trauma and I have learned that as this can affect my relationships it is better to be honest about it early. I don't go into too many details with the person, I simply state that I need a lot of 'alone time' and that I need to feel relaxed and secure to open up with someone else. If someone can't cope with it, it is perfectly fine, they are just not the right person for me.
There should be local charities that offer counselling and support for people with anxiety. Have a look at your local MIND for example.