LadySybilDeChocolate I think it's because they have the ability to take your children away -and- Colditz it stems from them being the deciding force that take your children. Even incapable, neglectful, and outright abusive parents still love and miss their children
For the umpteenth time - NO Social Worker can remove a child - it is a decision made SOLEY by a Court of Law, and with paperwork and evidence to support the Social Work request (as that's what it is) which would make your eyes bleed. I'm sure that those abusive parents do miss their children, would never suppose otherwise, but I still believe their child has a right NOT to be abused, emotionally, physically, or sexually, that overrides what their parents think/feel/believe.
Sakura and yet not possible to remove the abuser himself from the home . In Scotland there is recourse to removing an alleged abuser (Matrimonial Homes (Family Protection) (Scotland) Act 1981).
Sakura I don't know that much about SWs, but am I right to think you only need an undergraduate degree to be one? ONLY a four years Honours Degree, with a Diploma in Social Work gained concurrently, and practical, failable, placements - not that skilled then, eh?!
VivaLaSativa (Love your name) a significant number of sw's are middle classed, blinkered numpties that would rather do paperwork and lecture people than do something practical and actually help - only the Bosses to be fair 
To answer Charlotteperkins OP, the majority of my colleagues have been female, managers have been historically male. Not a great balance in my opinion.
I DESPAIR at the state of my profession, I really do...Damned if we do, damned if we dont - too fecking right 
...and in Scotland we also have the added layer of the Children's Hearing System - three (volunteer) lay members of the Public (and Oh so Middle Class worthies in many cases) who at a yearly Review Hearing can reject the recommendation of a Social Worker that has seen, visited, worked with, a child and their family, week in week out, and whom the Panel Members have never set eyes on...at least the guidelines for that process recommend a mix of genders 