I'm just the same. I've managed to trace the connections for my ancestors.
My experience may be totally different to yours, but maybe not.
Basically, most of my family have spent the last 400 years living within about a 20 mile radius of one city in the west of England. The men have mostly been farm labourers (although there was the occasional butcher, baker and stocking maker [also, an actual candlestick maker] - and even a rabbit catcher) and those women who were recorded as having jobs were generally laundresses or worked in domestic service.
So it's interesting finding out about the ones who left.
One example is two cousins who went over to America in the 1650s. They both ended up growing tobacco in Maryland.
Then, tracing their descendants, you are suddenly hit with the reality of slavery.
The first US Census was in 1790 and, at that time, as an example, one of my relatives in Maryland was recorded as including the following in the home:
Free White Persons - Males - Over 16 - 2
Free White Persons - Males - Under 16 - 2
Free White Persons - Females - 4
Number of Slaves - 7
So, that is two white men over 16, two white boys under 16, four white women and seven slaves.
It's interesting to note that "white males" are split between adults and children but women aren't. Neither are slaves.
It was not unknown for female slaves to get pregnant to their white masters.
This may well be where the distant relations come from.
'
'
But it could equally come from much later emigrations. For example, I also have relatives who emigrated to the US in the 1850s. Some of them then moved to Utah and became Mormons (or already were). Others moved to, as it was then, the Dakota Territory. Any of these could also be a source of the connection.
'
'
If you're wondering why links going back so far might show up in your DNA Matches, this is due to something called "endogamy".
In genetics there is something called "endogamy" which is where the same extended families tended to intermarry over several generations. You can then still see the results of this in the DNA hundreds of years later.
The intermarrying means that more of the dna is retained within the family for much longer and it makes it appear as though you are more closely related to somebody else than you actually are.
I do have connections with early US colonial families. When talking to distant relations, who are descendants of these families and are also studying their family tree and DNA, they have told me that they have found literally dozens of examples of finding ancestors on eg their father's tree who has a DNA link to their mother or that both their mother and their father have a DNA link to the same person.
Another thing with endogamy is that you end up with many more matches and also a lot of shared matches, precisely because families intermarried so much.
It is not uncommon for colonial Americans, Ashkenazi Jews, French Canadians and US Cajuns to have large numbers of DNA matches due to endogamy many decades or centuries previously. The same thing applies to certain communities from South Asia as well.
I've also had some of my extended family here in the UK marry each other over several generations in the 1700s and 1800s which also adds to this.
'
'
So TL;DR it's likely that you have some distant relations that went over to America and they may, or may not, have been slave owners.