I would say to your wife that you're sorry things have got where they are, you can see where you could have done more to support her in the past but from now on you will respect whatever boundary she needs to draw between herself and your family and you as her partner and their son will deal with the consequences.
It's perfectly reasonable for her to say she doesn't want them in the house. Your home is a safe place.
Especially as they live so close and you are able to take your daughter to see them at theirs there is no need for your wife to be forced to tolerate them in her own home.
This set up may not be a forever thing, if everyone is respectful then these boundaries may ease in the future and your wife may be able to grit her teeth through a visit but for now it seems like what needs to happen.
As for how often to see them, if you work 5-6 days a week and they are demanding/expecting your time on your (sometimes one) day off I think that's also completely disrespectful to you and your wife as parents of a young child and a family unit in your own right.
Support your wife.
Take your DD to see your parents every other week or so.
Make it clear that this is a consequence of their behaviour towards your family, not the fault of your wife and contact will likely be reduced even more if they continue to treat your wife and you the way they have been, then back that up if needs be.
They are not being treated differently because they are 'paternal grandparents'. They are being treated differently as a consequence of their own actions.
No one can dictate childcare arrangements for anyone else's child and they have no right to be offended by you and your wife's choices regarding this.
As for the PP who said a man would never be able to set that boundary, if my parents were ever rude to my DH I would absolutely pull them up on it and things would not be continuing until they apologised. If it was so bad he asked them not to come to the house I would absolutely respect that and take our DC to see them on my own. I don't understand how anyone can think that is not reasonable.
My PIL have been awful to me and my DH has not been the best at managing it. I would dearly love to withdraw myself from the situation and allow them to continue a relationship with their son and grandkids that doesn't involve me as much, but they live so far away they 'have' to stay with us for days on end to be able to see our DC.
I get trapped being treated appallingly in my own home. I have to be polite and host people that humiliate me.
I do it as I feel otherwise I'm obstructing their relationship with their DGC.
It does not do our marriage or my sense of well-being any good. It causes a huge amount of unhappiness every time. It's a shit situation.
Let your wife step back. Let her have a boundary. Facilitate a relationship between your DD and your parents that is sustainable and fair.
Take a hard line and pull your parents up on bad behaviour every single time or this will only get worse and will become a never ending source of strain in your marriage.
Good luck.