Dont make any hasty decisions and give yourself a bit of time. Its disappointing that the SW has decided to support his approach not yours.
I would not even bother to take your mum home. Your brother can organise a taxi.
If she is in somewhere supervised the burden does not fall on you. If she is at home the assumption seems to be that you can be relied on to pick up any pieces.
I would simply tell your brother that you appreciate his willingness to take over the responsibility. Given you both have different approaches and given you are tired, you welcome the break. Easier if he now takes over all the day to day stuff. Including organising the payment of bills.
I would not resign the POA. You will want it should your brother not step up to the mark.
I agree also about allowing your brother to sort out memory clinic attendance. I suspect they wont be thrilled to hear that SS have decided to follow your brother's 'informed' view.
So first thing. Do not pick your mum up. This is for your brother to do. He needs to arrange there is fresh food, that her fluid intake is supervised etc. Rule 1 is neither he nor SS should expect you to run around. They have decided that your mother is safest at home. You disagree, and feel this action will put your mum at risk. Hence you are not prepared to enable.
Also if there is a problem and your mum calls you perhaps you say that you will alert your brother.
What a mess. I am very shocked by the Social Services approach. They are meant to be safeguarding your mother. But then we are still waiting the discharge assessment for my mum from last December. It took a lot of chasing and two missed appointments before they turned up and now a couple of months later we have still to get the report. If she had not had family and cash my mother would have been sent home to an unheated flat in a confused state with a broken hip, no adaptations, no care plan and no means of shopping.
Given our mothers live in the same part of the country perhaps I should not be that surprised.
Still brace yourself. You are out of the loop. Take time off. Rest and recover. Dont enable.