Buttons - SW DO try to draw conclusions that only trained psychologists should be making. But they will also be doing that about your Ex as well.
Some of my experiences with SW's haven't been great, but I was in the care system as a child, and I think that coloured their perception of me as a parent, at least for the first few years, as I was very young when I had DD (still 16, still on the at-risk register myself).
More recently, most of my experience with SW's has been when Ex-P has made spurious complaints to them. They came out, inspected the house, told me what they were unhappy with, I jumped through their hoops, they then produced a report stating they have no problems with the way I look after my dc. Just with how often I clean my toilets. 
. Because once a day obviously isn't often enough. I have had no further dealings with them in the last 2 1/2 years, since just before I got back with Ex-P, but now that we have split up again, and I am starting to separate myself from him, I know it won't be long.
DON'T talk too openly with them in future about your emotions, they do not have the time or the training if they are Child Protection to care about YOUR feelings. It's a bit shitty, but if you need to talk, get a counsellor or chat on here, not to SW's.