Yes, it is detrimental to your kids. Children who are used to (known and trusted) family/friend babysitters, are happy to be fed/bathed /story and put to bed by them; if they wake, they are relaxed and confident to be soothed changed, given a drink and settled back to sleep.
Because they have that experience and familiraity with a range of adults, those children can cope with emergency separations from their parents; a work absence, car breakdown, illness, hospitalisation (of the parent or the child). The unforeseen circumsatnces that would be hugely more stressful for a child that's never had a sleepover at Granny's, never had a babysitter.
When they start nursery, and school, it's not scary to be left for the very fisty time ever ; they can take it in their stride because their parents have already expanded the social circle of "safe people I can trust, who are going to look after me... and I know mum will come back, like she always does".
I'd suggest it's also detrimental to the parents adult relationship. You are not JUST parents; you are adult partners who need to cultivate your relationship to keep it fresh and healthy. Do stuff on your own as a couple, go out to dinner, the cinema, a party, for a moonlight walk, to th gym , to see friends, share hobbies. Be able to go to a work dinner and stay overnight; have a relaxed weekend away together.