Historically, yes, I do think so, although realistically the period where there were a fair number of non-white actors looking for parts only started in the mid 20th century. Before that, in a lot of cases there wouldn't have been actors available to play the few parts that were intended to depict people from outside of Europe.
But once you get into the 1960s and through the 70s I certainly think there were a lot more actors who struggled because they just weren't what the people casting wanted because they were narrow in their thinking. Once you get to the 80s it seems to me that things began to open up, but it still took time.
I would say there is a pendulum swing element, people seem to have a tendency to think - of a diverse cast is good, so more is better. That type of thing usually corrects itself IMO.
What I really dislike is the tendency for some to say, well, white actors had their chance and others had a hard time, so it doesn't matter if George here is being treated unfairly now, somehow that compensates for the fact that white males had an advantage before. Unfairness in the present does not, whatever people like Kendi say, balance out unfairness in the past.