As everyone has pointed out. 4th cousin isn't a big deal. I found out that my husband and I were 10th cousins and that was after we had been married for 25 years with 2 sons.
Granted it isn't 4th cousins but I had that same response that you did. 
Come on who marries their cousin? How embarrassing. Then figured "what the hell!" I just didn't care. He is my soulmate and love of my life.
But when you look into small town/villages as you go further back in time, the late 1800s and further back to the beginning of time.
Remember small towns were still small towns. They didn't have a lot of people moving in and out of the towns like we do now. back then with just horse and wagon for transportation, so moving on a whim rarely happen. Instead couples had children, then children coupled up and had their own children and repeat.
Keep in mind the pond is small at the beginning and though it grows as generations come in. Add in how we treated each other at the time (still do to some extent) where Irish married Irish, Italians married Italians, religion married within their religion and so on. So even though the pond has been made smaller. I'm sure we all know that immaculate conception's only happened once.
That being said you have nothing to worry about. However, if your still concerned and still want to stay together but worry about how it will affect any children see a specialist and have genetic test run.
Just an interesting fact. Back in the 1930s, it was in this culture of heightened awareness (and misinformation) that states began to pass laws requiring couples to submit to blood tests before applying for marriage licenses, so they could avoid spreading a previously undetected venereal disease. At the time it was syphilis that was spreading like wild fire.
After the syphilis crisis was over, some states simply pivoted to using premarital blood tests to check for other diseases, like tuberculosis, rubella, and HIV. The problem, however, was that the practice didn’t actually uncover that many cases of any kind. The Mises Institute reported that the nation as a whole spent around $80 million on premarital syphilis tests and found only 456 positive cases; and according to a 1989 study in the Journal of the American Medical Association, prospective newlyweds in Illinois spent $2.5 million to test for HIV during the first six months of the program, and only eight of the 70,846 tests came back positive. Since neighboring states saw an increase in marriage license applications during that time, the study suggested that people were simply crossing state borders to avoid getting tested (after all, Illinois didn’t pay for the tests).
As states started to realize that premarital blood testing wasn’t a cost-effective way to screen for diseases, they abolished their laws. But it was definitely a slow process—Montana became the final state to repeal its mandatory blood testing (for rubella) in 2019.