AIBU to think this is turning unpleasant
If you subjectively feel it is unpleasant, then it is unpleasant for you!
YANBU to think I’m dealing with a toxic person and I should drop the client
Dunno. Only if you think her she was deliberately working in tandem with her friend, & they had a cunning plan to 1) distract you with chat 2) then spring the friend's criticism on you as part of a knowing sabotage.
And that would be a paranoia too far, no?
How about a middle ground, where you communicate with your client to find out what's going on, then find a way of working you can both be comfortable with?
Next time she asks you to sit down & make conversation:
"Client, I'm here to clean, & stopping to chat means I don't get to do what you are paying me for. We spoke for an hour last time! - so while it's nice to catch up with you, if we keep stopping to chat, I need to know that you are happy to extend the hours I'm here so that I can get your work done."
ie you are opening the door to a conversation about paying for your time - whether that time is spent in chat, or active cleaning.
If that's something you're happy to do - win/win. You get an extra PAID hour, your client gets a clean house and her conversation.
If you are not happy to do that, you need to either shut down the chat, or dump the client. Don't go down this rabbit hole of suspicion - she's obviously just looking for companionship. You simply need to make it clear what your terms are. E.g. if you usually clean for 3 hours at £15/hour, that's £45.
If she wants to continue the 3 hours PLUS an hour's chat, that's £60 of your time.
Don't muddle companionship/attempts at friendship with professionalism & billing. Be clear what you are offering, & if both are you are happy - great. If one of you is not - quit.