No, Mr and Mrs Crawley are not related to Violet.
It must have gone something like this:
Third Earl has a younger son who has to go off and make his living in a profession. Over the years this branch of the family loses touch with the ones who live at Downton. This chap's grandson or greatgrandson is Matthew.
The Third Earl's eldest son becomes the Fourth Earl. He has two surviving sons.
The younger one has a son (Robert's cousin) and a grandson who go down with the Titanic.
Meanwhile the elder son becomes the Fifth Earl. He marries Violet and has two children, Robert and Rosamond. Robert becomes Sixth Earl.
Robert has three daughters but can't leave them the estate and the title, so his heir is his cousin, next heir is cousin's son (Patrick?). Since they both die, they have to go back up the family tree and down again till they come to Matthew.