Vaccines are one of the most scrutinised and studied areas of medicine - the idea that "vaccine damage" might be much higher than "they" think is unfortunately a persistent antivax myth. It is not especially helped by some doctors being immediately dismissive or even impatient about people coming with genuine fears that something might be linked, but all suspected vaccine-linked issues are able to be reported independently of your individual doctor's say-so. The chicken pox vaccine has been standard in most countries for 10-15 years now, and up to 30 in some others, with no widespread problems occurring. In fact the main reason why it's difficult to pin down exact numbers for people harmed by vaccination is because it so rarely happens. It's a little bit like cot death in that respect (now so low it's almost impossible to advance the research and get any more information about why it happens).
One of the most common fears linked to vaccination is this nonspecific brain injury one - part of that we know the cause, it is acute allergic encephalitis, and doctors/nurses administering vaccinations have to be trained in how to react to this and is why it is important to keep an eye on your child for the 24/48 hours following a vaccination. Unfortunately it's not possible to reverse every case but in most cases it is. Either way this is very rare. The other regression-of-skills version is related to the usual profile of several disorders including autism spectrum disorders and some chromosomal disorders (e.g. tiny deletions) which technology has only recently been improved/clear enough to identify. These patterns occur in children with these types of disorders whether they were vaccinated or not, and not in any concordant kind of pattern relating to the timing of a vaccination. It must be extremely distressing to witness as a parent, and I can totally understand why vaccination would be under scrutiny as a potential cause - you'd want to investigate any possible cause in the manner of Dr. House sending his students to rifle through a patient's home and last known movements - but the link with vaccination has been suggested a lot and never found to have any compelling reason to link with cases like these. You might as well blame brand of washing powder. It's not logical to assume that vaccination is the one area where doctors/scientists are wrong, unless you have something specifically against vaccination, and I think two things feed in here - a totally natural revulsion/horror of the idea of pain caused by needles and the idea of putting a foreign substance directly into a child, not moderated through something like the digestive system (is it just me or do oral vaccines always somehow "feel" safer?) and the question of profit - alternative health traders have much to gain (selfishly) from the idea of discrediting "mainstream" medicine. Probably not small scale homeopaths, naturopaths, crystal healers etc - most of who seem to genuinely believe in what they sell, but follow up to the highest point of those profit triangles and sadly you find money and greed driving at least some of the antivax movement.
Then the second most common fear I have come across is the idea that vaccines are not some kind of singular trigger/cause of harm but that they are simply a small part of the pollution we as humans encounter every day, and can avoid some of by eating organic, avoiding plastics, reducing carbon use and so forth. The argument here is that if you are doing everything you can to avoid environmental pollutants in your life, you should also avoid them in medicines and vaccines, getting only those which are strictly necessary, or some argue none at all (possibly because they believe them to be inefficient). But again, this is a poorly thought out argument. The fact is we come into contact with a great many environmental pollutants every day. It's true that our modern lifestyles are harming us. But if you want to cut down on things which will harm you, it makes sense to approach it as a risk/benefit decision. In fact the amounts of these kinds of things included in vaccines are extremely small, much smaller than some tolerance levels allowed in food or things like car exhaust fumes, which we encounter every day. A vaccination on the other hand is something you will have once or a handful of times in your life, and has a huge benefit in terms of protecting you from terrible side effects of some diseases. If you want to cut down on environmental pollutants, this is a terrible way to do so. It would be similar to saying that a fridge is very harmful to the ozone layer (true) so you're going to store your meat and cheese at room temperature, not making any effort to keep it cool enough to reduce bacterial growth. In avoiding one risk you have increased another which is far more likely to have a direct effect on your own health and that of any visitors you feed.
The one certain downside of vaccination is the delivery method - you get a sore leg or arm. It's not especially pleasant but probably in the scheme of things no worse than being ill for a week.