What your mum has done is given a Personal Equity Transfer to both you and your brother out of her assets. It will fall within her estate if she dies within 7 years of gifting it and may be liable to inheritance tax depending on the size of her estate. The gift with reservation thing is a red herring and is more for people who put their homes in their children's names and then continue to live in it. In your case your mum has moved into your house. The fact she sold hers to give the gift is not relevant.
Now to the crux of your issue I had friends who did exactly as you did. The MIL died and within a few months our friends had moved the FIL into a house with an integral granny flat (self contained but also part of the main family house). The relationship with FIL before MIL died was problematic but became so much worse when he moved in with them. It almost broke their marriage, drove my friend to a nervous breakdown and they would both stay at work as long as possible rather than go home and have to cope with him. For anyone else reading this my advice would be to give it time before going down this route with newly bereaved parents or PIL and I think OP would probably agree now. Eventually as he got older he ended up going into hospital for a fall and they refused to have him back so he ended up in warden controlled sheltered accommodation and they supported as best they could but with boundaries.
I think setting boundaries would work in this case. Call her bluff next time she says she wants the £80k back and say no it was a gift in legal terms and she has had 4 years of living in your house so she can consider it rent. You cannot sell part of your house and on its own the two rooms will probably not add that much in value and you lost part of your land to build it. The money given to your brother is irrelevant but I think it is quite fair to say she gifted him almost the same as you without strings whereas your "gift" definitely had strings as you had no choice as to how it was spent, no time limit on when it can be realised and you sacrificed your personal privacy along with a degree of responsibility for your mums well being when you took it.
If she wants to use the main kitchen or the garden then she has a responsibility to help out in keeping them tidy and clean. If she won't help you out with babysitting or collecting the children from school then fine you won't be giving her lifts or do other things for her as she gets older.
£150 is not much towards bills if you are having to pay for heating through the day when she is home and you are out. She has to pay a third or a quarter of energy and water costs and council tax along with the cost of TV, internet, telephone etc. Add them all up and show them to her. Are the two rooms a self contained annexe and can they be metered separately if she protests?
It is difficult to see how this will end well given your relationship was not that great before you embarked on this and she seems to be a bit of a matriarch. In the end though you hold the cards. She is living in your home and if she does not like it she can leave.