You are not being unreasonable in questioning it.
I 100% believe in Karma and my understanding/belief is as follows.
To believe in Karma, you have to believe in the Soul and Reincarnation.
The three of them work together.
What we are born as and what happens in this life mainly depends on accumulated Karmas from previous lives.
There are some rare cases of 'instant Karma', but just like it takes time for fruit to ripen, it takes time for Karma to give its consequences, and one life is too short for that.
As you say, we see people who do terrible things but seem to get away with it, while others are good but struggle endlessly.
But we also see people who are good and not struggling and people who are bad and are struggling.
With a firm belief in Karma, we can connect these situations.
Those struggling endlessly, are actually paying for the terrible things they did but got off with in previous lives.
Those having it good are benefiting from good Karmas from previous lives.
Ultimately, it is a matter of faith and everyone is free to believe what they like.
It is impossible for anyone to prove to anyone else that Karma and/or Soul and/or Reincarnation are true or not.
Any talk of 'scientific proof' or 'scientific evidence' in this matter is therefore utter nonsense.