It's not so much that they are bad for you but they are just unnecessary. Soap and water will remove non resident flora ie. dirt whether that is dog poo, food, mud or whatever.
Antibacterial products are deigned to remove resident flora, bacteria that normally live on your skin, good for you bacteria if you like.
They are used in hospitals to remove this flora when nurses and doctors need to perform procedures where these "good for you" bacteria can cause harm, ie sterile procedures such as surgery, wound care, catheter insertion.
Also it is worth knowing that antibacterial soaps do not work as effectively as you might think. Many antibacterials do not work when used against organic matter, visible dirt. You are supposed to wash your hands first and then use an antibacterial product in order to get maximum effect.