You mention being scared of conflict. This can be a big source of failure to assert yourself and thus opening yourself to be mistreated more than the once.
I think of conflict as a major disagreement lasting for days that has no resolution, or a shouting/screaming/violent interaction. It is wise to avoid such conflicts unless essential.
I have known people who think of a conflict as another person feeling even slightly pissed off with them. This is a recipe for a life of martyr's misery
Try this though experiment. Someone wants you to do them a favour. Not life or death. It takes up your time or effort though. Like giving them a lift to something or rearranging plans. You could do it but really don't feel like it. The other person has a tendency to be more of a taker than a giver.
Test 1: can you imagine yourself saying "No, sorry, that doesn't work for me. Hope you work something out." Then you let them exit the conversation to go off and find alternatives. From you: no elaborate excuses, no saying yes, no offering other solutions, no keeping them in conversation for validation of no hard feelings. Can you even imagine such an ordinary interaction?
Test 2: you do (1). The person responds by saying "No worries, see you on Monday." Would you then forget about it, maybe asking on Monday what they did about that situation, or would you agonise for hours about saying no, analyse every message/silence from them, start overcompensating like mad?
Test 3: instead of saying no worries what if the person looked disappointed and said "Oh no, I was relying on you. I have already arranged for X. Woe woe is me. Please help me." Can you imagine yourself saying "Oh that's a shame. Oh well, I'm sure you'll think of something. Good luck!" Or would you capitulate immediately. Would you agonise about their feelings forever and a day? Would you feel like a cowbag? Or would you think "Cheeky mare taking me for granted like that and then having the cheek to tell me she's done it and then try to lay guilt on me. I'll have to keep an eye on favours for that cheeky cow."
None of that was conflict in my opinion. If you can't do those kinds of interactions then you are setting yourself up to be trampled upon. If so, go get some assertiveness training.